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7th  Annual  CMMI  Technology  Conference  &  User  Group 

“ Investigation ,  Measures  and  Lessons  Learned  about  the  Relationship  between  CMMI®  Process  Capability  and 

Project  or  Program  Performance ” 

Denver,  Colorado 
November  12  -  15,  2007 


MONDAY,  NOVEMBER  12,  2007 


•  CMMI  VI  .2  —  An  Overview  Mr.  David  Phillips,  SEI 


TUESDAY,  NOVEMBER  13,  2007 


State  of  CMMI® 

•  Mr.  Clyde  Chittister,  Chief  Operating  Officer,  SEI 
Executive  Panel 

Panelists: 

Ms.  Kristen  Baldwin,  Office  of  the  Secretary  of  Defense 
Mr.  Tom  Neff,  Defense  Threat  Reduction  Agency 
Mr.  Rich  Frost,  General  Motors 

Lunch  with  Guest  Speaker 

•  Mr.  Mark  Schaffer,  Director,  Systems  &  Software  Engineering,  OSD  (AT&L) 

Technical  Sessions 

TRACK  1 

•  When  the  Only  Tool  You  Have  is  a  Hammer,  Every  Problem  Begins  to  Look  Like  a  Nail,  Mr.  Sam  Fogle,  ACE  Guides,  LLC 

•  The  Journey  to  CMMI  Level ,  Mr.  Andrew  Lay,  Lockheed  Martin  Aeronautics  Company 

•  Visualizing  Improvement  with  Capability  Waypoints,  Mr.  Robert  Jacob, 

Naval  Air  Systems  Commandlnstitutionalization  Measures:  Key  to  Improved  Process  Monitoring,  Dr.  John  Rusnak,  Lockheed  Martin  Space  Systems 
Company 

TRACK  3 

•  Assuring  Quality  for  Efficient  &  Sufficient  Testing  Mr.  Pramod  Varma,  Wipro  Technologies 
TRACK  4 

•  Bridging  Process  Improvement  During  Program  Management  Evolution:  An  Experience  Report  Capt  DeWitt  Latimer,  USAF 

•  An  “Embedded  SCAMPI-C”  Appraisal  at  the  National  Security  Agency.  Mr.  Joseph  Wickless,  SEI 

TRACK  5 

•  Linking  Project  Performance  to  CMMI  Process  Capability  through  Lean  Measurements,  Mr.  Jeffrey  Dutton,  Jacobs  Technology 

•  Quantitative  Models  for  Predicting  Project  Success,  Dr.  Rick  Hefner,  Northrop  Grumman  Corporation 

TRACK  6 
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•  How  to  Kick  Start  a  Process  Improvement  Effort  to  Achieve  a  CMMI  Rating,  Ms.  Brenda  Hall,  Computer  Sciences  Corporation 

•  SEI  Appraisal  Program  Quality  Report,  Mr.  William  Hayes,  SEI 

•  The  Process  In-execution  Review  (PIER)  After  Three  Years,  Mr.  Dale  Swanson,  The  MITRE  Corporation 

•  I’m  Preparing  My  Organization  for  an  Appraisal,  but  I’m  Not  Really  Sure  I  Understand  this  PHD  Thing.  Should  I  Worry?,  Mr.  Sam  Fogle,  ACE  Guides,  LLC 
TRACK  7 

•  Aligning  CMMI  and  ITIL  -  Where  Am  I  and  Which  Way  Should  I  Go,  Mr.  Pat  Mitryk,  Cognence,  Inc 

•  Integrated  System  Framework:  A  Way  Out  of  the  Multi-Model  Madness,  Mr.  Paul  Byrnes,  Integrated  System  Diagnostics 


WEDNESDAY,  NOVEMBER  14,  2007 


Lunch  with  Guest  Speaker 

•  Ms.  Mary  Poppendieck,  President,  Poppendieck,  LLC 
Technical  Sessions 

TRACK  1 

•  CMMI  Contenders,  CMMI  Pretenders,  Dr.  Rick  Hefner,  Northrop  Grumman  Corporation 

•  Initial  Fears  of  CMMI  Introduction  and  How  Things  Really  Played  Out,  Dr.  Paul  Nugent,  General  Dynamics  Advanced 
Information  Systems 

•  Software  Firm  +  CMMI  Level  2  Initiative  +15  months  =  Dramatic  Quality  Improvements,  Mr.  Jeff  Simpson,  Campus  Management  Corporation 

•  How  to  Explain  the  Value  of  Every  CMMI  Practice,  Dr.  Rick  Hefner,  Northrop  Grumman  Corporation 

•  Mrs.  Doubtfire  Answers  Your  Questions  about  Process  Improvement,  Dr.  Rick  Hefner,  Northrop  Grumman  Corporation 

•  Developing  a  Second  Generation  Directive  System  Architecture,  Mr.  Kenneth  Weinberg,  Raytheon  Company 

•  Whose  Processes  Are  These,  Anyway,  Ms.  Judith  Tejan,  AAI  Services  Corporation 

•  Scientific  Breakthroughs  in  Process  Improvement,  Ms.  Cheryl  White,  Change  Delivery  Group 

TRACK  2 

•  The  What,  When,  Why  and  How  for  CMMI  Training,  Mr.  Tom  Bragg,  AVISTA  Incorporated 

•  Transitioning  to  the  CMMI:  What  They  Never  Told  You,  Mr.  Steve  Fried,  The  Boeing  Company 

•  CMMI  Implementation:  Overcoming  the  PPQA  Challenge,  Mr.  Pat  Mitryk,  Cognence,  Inc. 

•  How  to  Measurably  Improve  Your  Requirements,  Mr.  Timothy  Olson,  Lean  Solutions  Institute,  Inc.  (QIC) 

TRACK  3 

•  Using  Lean  Six  Sigma  to  Implement  CMMI  High  Maturity  Practices,  Ms.  Beth  Clark,  Lockheed  Martin 

•  The  Potential  for  Lean  Acquisition  of  Software  Intensive  Systems,  Mr.  Jeffrey  Dutton,  Jacobs  Technology 

•  Lean,  CMMI  and  Six  Sigma  Working  Together  to  Achieve  High  Success,  Ms.  Susan  Bassham,  US  Army  Aviation  and  Missle  Command 

•  Comparing  and  Contrasting  the  PP  &  PMC  Process  Areas  of  CMMI  v  1.2  and  SCRUM,  Dr.  Aldo  Dagnino  ABB,  Inc.  -  US  Corporate  Research 

•  Effective  Systems  Engineering:  What’s  the  Payoff  for  Program  Performance?,  NDIA  Systems  EngineeringsEffectiveness 

•  What’s  All  this  ‘churn’  in  Systems  Engineering  Standards  and  Models!?,  Mr.  Donald  Gantzer,  SAIC 

TRACK  4 

•  Driving  Process  Improvement  Using  the  CMMI-ACQ  at  General  Motors,  Dr.  Richard  Frost,  General  Motors 

•  Leading  Indicators  for  Acquisition  Programs,  Mr.  Robert  Ferguson,  SEI 

•  CMMI  High  Maturity  Misconceptions,  Mr.  William  Hayes,  SEI 

•  High  Maturity:  How  Do  We  Know?,  Dr.  Mike  Konrad,  SEI 

•  High  Maturity  System/Software  Cost  Estimation,  Dr.  Richard  Welch,  Northrop  Grumman  Corporation 

•  ADVANCE  -  Implementing  a  Defect  Model  for  Performance  Prediction,  Mr.  Stanley  Martin,  L-3  Communications/IS 

•  Statistically  Managing  a  Critical  Logistics  Schedule  Using  CMMI,  Mr.  Robert  Tuthill,  Northrop  Grumman  Corporation 

•  A  More  Practical  Set  of  High  Maturity  Practices,  Dr.  Rick  Hefner,  Northrop  Grumman  Corporation 

TRACK  5 

•  Program  Level  Return  on  Investment  for  CMMI®  Process  Improvement,  Mr.  J  Perry,  BAE  Systems 

•  How  Do  We  Get  on  the  Road  to  Maturity?,  Mrs.  Debra  Perry,  Harris  Corporation 

•  Understanding  CMMI  Measurement  Capabilities  Performance  &  Outcomes:  Results  from  the  2007  SEI  State  of  Measurement  Practices  Survey,  Dr.  Dennis 
Goldenson,  SEI 

•  Using  Predicted  Delivered  Defects  as  a  Management  Tool,  Mr.  Dustin  Sims,  BAE  Systems 

•  Calibrating  the  Project  Planning  Process,  Mr.  Donald  Corpron,  Northrop  Grumman  Corporation 

•  All  Others  Bring  Data,  Ms.  Charlene  Gross,  SEI 

TRACK  6 

•  Executing  a  Successful  CMMI  Maturity  Level  3  Scampi  for  Spawar  Systems  Center  Charleston,  Mr.  Michael  Kutch, 

SPAWAR  Systems  Center  Charleston 
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•  CMMI  SCAMPI  Appraisals  -  The  People/The  Process/The  Results-United  Space  Alliance,  LLC  Lessons  Learned,  Ms.  Robin  Hurst,  United  Space  Alliance, 
LLC 

•  Proposed  Approach  to  Heterogeneous  CMMI  Appraisals,  Mr.  Joseph  Vandeville,  Northrop  Grumman  Corporation 

•  Selecting  a  Representative  Sample  for  CMMI  Enterprise  Appraisals,  Ms.  Kathryn  Kirby,  Raytheon  Company 

•  Logistics  and  Lessons  Learned  in  Conducting  an  CMMI®  Maturity  Level  3  Full-Model  Scope  Enteprise-level  Appraisal 
Ms.  Kathryn  Kirby,  Raytheon  Company 

TRACK  7 

•  Excellence  at  the  Organization,  Team  and  Individual  levels;  CMMI,  TSP  and  PSP  -  Experience,  Lessons  Learned  and  Why  all  Three  are  Needed,  Mr.  Girish 
Seshagiri,  Advanced  Information  Services,  Inc. 

•  IEEE  Life  Cycle  Standards  and  the  CMMI®  -  Implementation  Considerations,  Dr.  Peter  Hantos,  The  Aerospace  Corporation 

•  Using  CMMI  and  OPM3  to  Improve  Performance,  Mr.  Thomas  Keuten,  Pariveda  Solutions 

•  Complementary  or  Competing?  Achieving  Synergy  with  OPM3®,  CMMI®,  and  ISO  9001-2000,  Mr.  Mark  Scott,  Harris  Corporation 

•  Formal  Process  Definition  with  Industry  Standards,  Mr.  Chris  Armstrong,  Armstrong  Process  Group,  Inc. 

•  Project  Management  Architecture  Design  as  a  Critical  Success  Factor  in  CMMI  Model  Implementation,  Mr.  Christen  MacMillan,  L-3  Communications 


THURSDAY,  NOVEMBER  15,  2007 


Lunch  and  Award  Presentation 
TRACK  1 

•  Fast  Track  to  Higher  CMMI  Maturity  Levels:  Lessons  Learned  from  Five  Initiatives,  Ms.  Cheryl  White,  Change  Delivery 

•  Seven  Success  Factors  for  CMMI  Based  Process  Improvement,  Mr.  Orhan  Kalayci,  XPI  -  extreme  Process  Improvement 

•  CMMI  Process  Improvement:  It’s  Not  a  Technical  Problem,  It’s  a  People  Problem!,  Mr.  Rolf  Reitzig,  Cognence 

•  Improving  Project  Proposal  Quality  via  CMMI,  Mr.  Chen  Wang,  Institute  for  Information  Industry 

TRACK  2 

•  A  Framework  to  Manage  and  Evaluate  Remote  Software  Testing  Using  CMMI,  Dr.  Aldo  Dagnino,  ABB,  Inc.  -  US  Corporate  Research 

•  CMMI,  Configuration  Management,  and  Baseball  -  How  to  Score,  Ms.  Julie  Schmarje,  Raytheon  Company 

•  Automated  Systems  for  Project  Portfolio  Management  -  Project  Success  and  Outstanding  Earned  Value,  Mr.  Pothiraj  Selvaraj, 

Global  Computer  Enterprises 

TRACK  3 

•  Project  Management  by  Functional  Capability,  Mr.  Fred  Schenker,  SEI 

•  Software  Architecture  Development  Leveraging  the  Attribute  Driven  Design  and  CMMI  Methodologies,  Dr.  Aldo  Dagnino, 

ABB,  Inc.  US  Corporate  Research 

•  Systems  Assurance  -  Practices  Make  Perfect  -  How  Your  Engineering  and  Management  Practices  Can  Help  Meet  the  Assurance  Challenge,  Mr.  Paul  Croll, 
Computer  Sciences  Corporation 

•  Tools  and  Resources  to  Enable  Systems  Engineering  Improvement,  Mr.  Michael  Kutch,  SPAWAR  Systems  Center  Charleston 

•  Applying  CMMI  Principles  to  Certification  Process  of  Legacy  Aircraft,  Ms.  Michele  Bruno,  The  Boeing  Company 

•  Accreditation  of  Undergraduate  Programs  in  Computing,  Software  Engineering,  Systems  Engineering  and  the  Ties  to  CMMI-based  Improvement,  Mr.  Dan 
Nash,  Raytheon  Company 

•  How  Future  Trends  in  Systems  and  Software  Engineering  Bode  Well  for  Enabling  the  Rapid  Adoption  of  CMMI,  Dr.  Ken  Nidiffer,  SEI 
TRACK  4 

•  Thought  Before  Action:  A  High  Maturity  Roadmap  for  the  Lower  Maturity  Organization,  Mr.  James  McHale,  SEI 

•  Integrated  Implementation  of  Advanced  Maturity  Practices,  Mr.  Dale  Childs,  DFAS 

•  Process  Performance  Baselines  and  Models:  Duh,  I  Don’t  Get  It,  Ms.  Diane  Mizukami- Williams,  Northrop  Grumman  Mission  Systems 

•  Expanding  Statistical  Process  Control  Across  All  Engineering  Disciplines:  A  Sequence  of  Practical  Case  Studies,  Dr.  Richard  Welch,  Northrop  Grumman 
Corporation 

•  Statistical  Process  Control  Applied  to  Specification  Requirements  Process,  Mr.  A1  Florence,  The  MITRE  Corporation 

•  Implementing  High  Maturity  in  a  Production  Support  Environment,  Ms.  Virginia  Slavin,  SSCI 

•  Using  the  Scientific  Method  at  Levels  4-5,  Dr.  Jeff  Ricketts,  Raytheon  Company 

TRACK  5 

•  The  Productivity  Puzzle,  Mrs.  Jill  Brooks,  Raytheon  Company 

•  Using  Metrics  to  Develop  a  Software  Project  Strategy,  Mr.  Donald  Beckett,  Quantitative  Software  Management 

•  Lessons  Learned  in  the  Implementation  of  Measurement  Techniques  for  CMMI  GP  2.8,  Dr.  Susanna  Schwab,  L-3  Communications 

•  Optimizing  the  Measurement  Process,  Mr.  Gary  Natwick,  Harris  Corporation 

•  Measurement  Strategies  in  the  CMMI,  Dr.  Rick  Hefner,  Northrop  Grumman  Corporation 

•  5  Major  Sites,  4  Separate  Disciplines,  1 1,500  Engineers,  1  Data  Repository:  Having  Data  You  Can  Actually  Use  -  Priceless! 

Mrs.  Jill  Brooks,  Raytheon  Company 

TRACK  6 
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•  Cutting  Appraisal  Costs  in  Half,  Dr.  Rick  Hefner,  Northrop  Grumman  Corporation 

•  Experiences  Implementing  Very  Large  High  Confidence  Enterprise  Appraisals,  Mr.  Paul  Byrnes,  Integrated  System  Diagnostics 

•  Process  Compliance  the  Smart  Way,  Mr.  Gary  Natwick,  Harris  Corporation 

•  Judging  the  Suitability  of  Alternative  Practices,  Dr.  Rick  Hefner,  Northrop  Grumman  Corporation 

•  Lessons  Learned  Conducting  High  Maturity  SCAMPIs,  Mr.  Paul  Byrnes,  Integrated  System  Diagnostics 

•  Benefits  of  SCAMPI  Class  C  in  Small  Settings,  Dr.  Mary  Anne  Herndon,  Transdyne  Corporation 

•  Lower  Cost,  More  Effective  Alternatives  to  SCAMPIs,  Dr.  Rick  Hefner,  Northrop  Grumman  Corporation 

•  Using  Workshops  to  Speed  CMMI  Adoption  and  Evidence  Gathering,  Dr.  Rick  Hefner,  Northrop  Grumman  Corporation 

TRACK  7 

•  Quality  Maturity  Model  -  Foundation  for  Process  Institutionalization,  Mr.  Sumit  Gupta,  Royal  Bank  of  Scotland  -  India  Development  Center 

•  Not  Just  for  Software  Anymore:  Lessons  Learned  From  a  CMMI™  Appraisal  on  Projects  in  a  Nonnuclear  Weapons  Facility, 

Mr.  Daniel  Fritts,  Honeywell 

•  CMMI  for  Services  Overview,  Mr.  Craig  Hollenbach,  Northrop  Grumman  Corporation 

•  Defining  Lean  Service  and  Maintenance  Processes  that  are  CMMI  Compliant,  Mr.  Timothy  Olson,  Lean  Solutions  Institute,  Inc.  (QIC) 

•  Implementing  Acquisition  and  System  Engineering  Processes  in  a  Maintenance  Organization,  Mr.  Bill  Fetech,  The  MITRE  Corporation 
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CMMI  is  registered  in  the  US  Patent  &  Trademark  Office  by  Carnegie  Mellon  University 


Conference  Agenda 


SUNDAY,  NOVEMBER  11,  2007 


3:00  PM  -  6:00  PM 

Conference  Registration  Open 

Grand  Mesa  Foyer 

MONDAY,  NOVEMBER  12,  2007 

The  Tutorial  sessions  require  a  $275  registration  fee  which  is  in  addition  to  the  Conference  registration  fee. 

7:00  AM -7:00  PM 

Conference  Registration  Open 

Grand  Mesa  Foyer 

7:00  AM -8:00  AM 

Continental  Breakfast 

Grand  Mesa  Foyer 

8:00  AM-  5:30  PM 

Tutorial  Sessions  (must  be  registered) 

Refer  to  Following  Page 

9:45  AM-  10:15  AM 

Break  (Tutorial  Attendees  Only) 

Grand  Mesa  Foyer 

12:00  PM  - 1:00  PM 

Lunch  (Tutorial  Attendees  Only) 

Grand  Mesa  ABC  Corridor 

2:45  PM -3:15  PM 

Break  (Tutorial  Attendees  Only) 

Grand  Mesa  Foyer 

5:30  PM  -  7:00  PM 

Reception  (Open  to  all  Attendees) 

Atrium  Display  Area 

TUESDAY,  NOVEMBER  13,  2007 

7:15  AM -7:00  PM 

Conference  Registration  Open 

Grand  Mesa  Foyer 

7:15  AM  -  8:15  AM 

Continental  Breakfast 

Grand  Mesa  Foyer 

8:15  AM -8:30  AM 

Welcome  &  Opening  Remarks 

Grand  Mesa  DEF 

♦  Mr.  Sam  Campagna,  Director,  Operations,  NDIA 

♦  Mr.  Bob  Rassa,  Director,  Systems  Support,  Raytheon 

Company 

8:30  AM -9:15  AM 

State  of  CM  Ml® 

Grand  Mesa  DEF 

♦  Mr.  Bob  Rassa,  Director,  Systems  Support,  Raytheon 

Company 

9:15  AM-  10:00  AM 

*  Mr.  Clyde  Chittister,  Chief  Operating  Officer,  SEI 

CM  Ml®  Into  the  Future 

Grand  Mesa  DEF 

♦  Mr.  Bob  Rassa,  Director,  Systems  Support,  Raytheon 

Company 

10:00  AM  -  10:15  AM 

Break 

Grand  Mesa  Foyer 

10:15  AM  -  11:45  AM 

Executive  Panel 

Grand  Mesa  DEF 

12:00  PM  -  1:30  PM 

Moderator: 

Mr.  Bob  Rassa,  Raytheon  Company 

Panelists: 

Ms.  Kristen  Baldwin,  Office  of  the  Secretary  of  Defense 
Mr.  Tom  Neff,  Defense  Threat  Reduction  Agency 

Mr.  Rich  Frost.  General  Motors 

Mr.  Mike  Phillips,  Software  Engineering  Institute 

Lunch  with  Guest  Speaker 

Grand  Mesa  ABC  Corridor 

♦  Mr.  Mark  Schaffer,  Director,  Systems  &  Software  Engineering,  OSD  (AT&L) 

1:30  PM -5:00  PM 

Technical  Sessions 

Refer  to  Following  Pages 

3:00  PM  -  3:30  PM 

Break 

Grand  Mesa  Foyer 

5:00  PM  -  6:30  PM 

CMMI-ACQ  Rollout  Reception 

Atrium  Display  Area 

WEDNESDAY,  NOVEMBER  14,  2007 

7:15  AM -5:00  PM 

Conference  Registration  Open 

Grand  Mesa  Foyer 

7:15  AM  -  8:15  AM 

Continental  Breakfast 

Grand  Mesa  Foyer 

8:15  AM-  11:45  AM 

Technical  Sessions 

Refer  to  Following  Pages 

9:45  AM-  10:15  AM 

Break 

Grand  Mesa  Foyer 

12:00  PM  -  1:30  PM 

Lunch  with  Guest  Speaker 

Grand  Mesa  ABC  Corridor 

1:30  PM -5:00  PM 

♦  Ms.  Marv  Poooendieck.  President.  Poooendieck.  LLC 
Technical  Sessions 

Refer  to  Following  Pages 

3:00  PM  -  3:30  PM 

Break 

Grand  Mesa  Foyer 

THURSDAY,  NOVEMBER  15,  2007 

7:15  AM -5:00  PM 

Conference  Registration  Open 

Grand  Mesa  Foyer 

7:15  AM  -  8:15  AM 

Continental  Breakfast 

Grand  Mesa  Foyer 

8:15  AM-  11:45  AM 

Technical  Sessions 

Refer  to  Following  Pages 

9:45  AM-  10:15  AM 

Break 

Grand  Mesa  Foyer 

12:00  PM  -  1:30  PM 

Lunch  and  Award  Presentation 

Grand  Mesa  ABC  Corridor 

1:30  PM -5:00  PM 

Technical  Sessions 

Refer  to  Following  Pages 

3:00  PM  -  3:30  PM 

Break 

Grand  Mesa  Foyer 

RECEPTION  (5:30  PM  -  7:00  PM)  (ALL  ATTENDEES) 
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NDIA  7th  Annual  CMMI  Technology  Conference 
&  User  Group:  12- 15  Nov  2007 
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ANDUSER/9R0UP 
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Enterprise  Architecture  Practice  Lead 
Responsible  for  corporate  IT  Policy  definition 


Key  roles 


Chief  /  Lead  /  Principal 

Strategist 

Mentor 

Auditor 

CMM-CMMI  adoption  background  in  high-maturity  (3-5)  organizations 
Worked  on  /  advised  resources  in 


Regular  and  recognized  speaker  and  writer  on  Enterprise  Architecture  and 
Systems  Engineering  Strategy 


CMMT"a 

Technology 

Conference 


D  USER  GROUP 
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By  a  leading  IT  research  and  strategy  consulting  firm 
In  early  2007 

As  EA  strategist,  TOGAF  expert  and  auditor  aware  of  the  concepts 
of  Capability  and  Maturity 

To  plan  and  lead  Architecture  Maturity  Appraisal  in  a  large  UK 
public  sector  organization 
T 

Endeavored  to  follow  TOGAF  as  its  EA  framework 
Had  ‘ heard  the  sound '  of  CMM 

Was  aware  of  the  Architecture-CMM  referred  in  context  of  US  DoC  and 
some  state  government  bodies 
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Bac  round  Wor 
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Had  EA-relevant  subject  list  (like  Process  Areas) 

Had  a  subject-based  capability  rating  scale 

Individual  ratings  were  associated  with  adjectives  (like  CMMI  levels) 


H 

Ratings  were  subjective 

Concept  of  Maturity  as  an  ‘ evolutionary  plateau  building  on  characteristics 
of  lower  levels'  was  absent 

Dependence  of  subjects  (PAs)  on  other  subjects  was  not  considered ,  though 
usually  obvious  to  EA  practitioners 

Scores  (usually  being  averages)  ended  up  being  fractional  thus  not  giving 
any  real  meaning  or  clear  adjective,  to  focus  improvement 
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of  interaction  /  process 
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Engage  more  closely  with  the  business  on  Architecture  Communication, 
including  Education  and  Communities 
•  •  • 

Look  for  patterns  in  business  processes  and  be  willing  to  develop  cloned 
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GM/Pariveda 

Tom  Keuten 

Defense  Materiel  Organisation 

Brad  Doohan 

Army 

John  Scibilia 

Defense  Acquisition  University 

George  Prosnik,  Larry  Baker 

SEI 

Mike  Phillips,  Mike  Konrad,  Aaron  Clouse, 
Roger  Bate,  Sandy  Shrum,  Keith  Kost, 
Rhonda  Brown 

Institute  for  Defense  Analyses 

Karen  Richter,  Margaret  Porteus 
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ed  Features 


Enhancements  to  CMF  PAs 


A  Measurement  and  Analysis 

i  Includes  earned  value  management  material 
i  Consistency  across  the  model  in  measurement  terms 

A  Project  Planning 

i  Includes  establishment  and  maintenance  of  a  project®  acquisition 


strategy 


A  Project  Planning  and  Project  Monitoring  and  Control 

i  Includes  important  specific  practices  on  transition  to  operations  and 


support 


A  Integrated  Project  Management  and  Organizational  Process 
Development 

i  Includes  expected  material  on  integrated  teaming 

i  Crucial  to  stakeholder  involvement  for  acquisitions  in  a  system  of 
system  environment 
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inded  Features 


i^^hlights  of  Acquisition  PAs 


ASolicitation  and  Supplier  Agreement  Development 
(SSAD)  and  Agreement  Management  (AM) 

i  Similar  to  Supplier  Agreement  Management  in  CMMI-DEV  but 
greatly  expanded  into  2  PAs 

T  Covers  both  legal  contracts  and  other  forms  of  supplier 
agreements  such  as  interagency  MOAs 

AAcquisition  Requirements  Development 

T  Similar  to  Requirements  Development  in  CMMI-DEV,  but 
develops  customer  and  contractual  requirements 

T  At  maturity  level  2  due  to  its  importance  in  acquisition 
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i^^hlights  of  Acquisition  PAs 


AAcquisition  Technical  Management 

V  Emphasizes  technical  reviews  and  technical  performance 
measurement  for  oversight  of  the  supplier 

V  Interface  Management  included  to  complement  the  other  kinds  of 
technical  management  process  areas  (e.g.,  Risk  Management, 
Requirements  Management) 

AAcquisition  Verification 

V  Added  peer  review  specific  goal  as  in  CMMI-DEV 

AAcquisition  Validation 

T  Applies  to  both  acquirer  and  supplier  activities,  similar  to  CMMI- 
DEV  Validation 


Rolling  ACQ  Out 
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AACQ  is  available  at  SEI  Website 

V  http://www.sei.cmu.edu/cmmi/models/index.html 

ADoD  views  CMMI-ACQ  as  a  source  of  best  practices  for  the 
Program/Enterprise 

V  DoD  encourages  a  continuous  implementation 

V  DoD  will  not  mandate  its  use  for  acquiring  organizations 

AThe  Acquisition  Module  (CMMI-AM)  is  still  available 

T  SEI  in  process  of  updating  it  to  reflect  content  of  the  ACQ 

ACMMI-ACQ  will  be  available  for  certified  appraisal  in  May  08 

AThe  model  and  training  will  continue  to  be  refined  as 
experience  is  gained 


I  APDF 

■»  Complete 


Your  complimentary 
use  period  has  ended. 
Thank  you  for  using 
PDF  Complete. 


and  Expanded  Features 


=  Software  Engineering  Institute 


High  Performance  versus 
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nd  Key  Message 


Achieving  a  high  maturity  rating  doesncb  guarantee  high 
performance 

Many  people  believe  that  when  you  achieve 
a  high  maturity  rating  that  high  performance  followse 

It  doesn<i>  always  work  that  waye 


A  You  can  get  high  performance  before 
you  get  to  high  maturity 


A  To  get  high  performance,  you  need  to 
build  a  solid  foundation  from  the 
beginning 
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cel  lent  Framework 


Offers  the  rWhatcft  what  needs  to  be  done  to  create  and 
improve  your  processes? 


Provides  a  commonly  understood 
framework  for  improving  organizational 
capability 


Does  not  provide  operational  processes 
(doesncb  provide  the  rHow^ 
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Are  we  getting  to  get  more  business  moving  to  a  higher 
maturity? 

Are  we  shipping  (releasing)  higher  quality  products? 

Do  we  have  better  performance? 

Do  our  products  have  more  functionality? 

Are  we  reducing  our  costs? 

Are  we  meeting  our  schedule? 

How  do  we  get  high  performance  from  high  maturity? 
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High  Performance  i 

High  performance  means  obtaining  superior 
outcomes. 

High  Maturity  T 

Implementing  the  concepts  and  practices  at 
levels  4  and  5  of  CMMI. 


High  Maturity  Practices  T 

The  "specific  practices"  and  "generic  practices"  at  levels  4  and  5  of  CMMI. 
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he  “WHAT” 


CMMI  is  a  model  not  a  process. 

It  describes  the  characteristics  of  effective  processes. 


The  trick  is  to  translate  the  model 
into  implementation. 

How  can  you  do  this  effectively  and 
efficiently? 
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rformance,  Address  Team  and 
ipline 


A  high-performing  organization 
must  be  built  of  high  performing 
teams. 

High  performing  teams  must  be 
built  of  high-performing  individuals.  ®U1U1/ 

High-performing  individuals 
must  be  disciplined  to  gather 
and  use  their  own  data. 


llUfUVf  II  until  I 

111  III 


For  a  successful  case  study  showing  the  integration  of  CMMI  and  TSP,  please  see  rDMMI  Level  5  and 
the  Team  Software  Processoby  Webb,  Miluk,  and  Van  Buren  in  CrossTalk  April  2007. 
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to  Building  High  Performance 

Many  organizations  are  becoming  team-based  to  achieve 
high  performance. 

A  You  build  high  performance  capabilities  within  teams. 

A  Then  you  extend  team  performance  to  the  organization 
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rl  am  directing  a  fundamental  changee 
through  integrated  product  teamse  the 
potential  value  is  to  reduce  cycle  time, 
improve  quality,  and  reduce  cost  of 
acquiring  quality  products  and  servicese  6 

-  William  Perry,  former  U.S.  Secretary  of  Defense 
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e  Data  Come  From? 


“The  government  are  very  keen  on  amassing 
statistics.  They  collect  them,  add  them, 
raise  them  to  the  nth  power,  take  the  cube 
Root,  and  prepare  wonderful  diagrams. 

But  you  must  never  forget  that  every  one  of 
these  figures  comes  in  the  first  instance 
from  the  village  watchman,  who  just  puts 
down  what  he  damn  pleases.  ” 


(Sir  Josiah  Stamp  quoting  an  anonymous  English  judge.) 
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What  does  operationalize  mean? 

A  To  put  something  to  use 

What  are  characteristics  of  an  rbperationalizedoprocess? 

A  The  people  who  use  the  process  own  the  process  and 
have  the  authority  to  adapt  and  improve  it. 

A  The  rprocess  ownersoare  in  the  best  position  to 
understand  the  process  strengths  and  weaknesses. 

A  If  people  rbwn  the  process, othey  will  be  more  willing  to 
fairly  evaluate  process  changes. 
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TSP  Week  Summary  -  Form  WEEK 


Name 

Consolidation 

Team 

Example  week 

Status  for  Week 

* 

~V  2 

Selected  Assembly 

Week  Date 

7/2/2007 

SYSTEM 

Task  Hours  %Change 

Weekly  Datau 

Baseline 

1280.1 

Schedule  hours  for  this  week 

Current 

1332.1 

Schedule  hours  this  cycle  to  date. 

%Change 

4.1% 

Earned  value  for  this  week 
Earned  value  this  cycle  to  date 
To-date  hours  for  tasks  completed 

44.7 


To-date  average  hours  per  week  43.4 


31.9 


24.3 


EV  per  completed  task  hour  to  date  0.075  0.105 


Date 


1.40 


1.79 


11/8/2007 


Cycle 


Plan  - 
Actual 

18.6 

38.3 


Project  End 

Baseline  _ 

Plan 


Dates 

2/4/2008 

2/4/2008 


A  team  is  in  week  2  of  7  month  plan. 

The  team  is  behind  10%  in  Earned  Value  but  the  projected  date  for  project  completion  is  2  years  lateS 
what  is  the  problem? 

The  team  on  average  is  only  getting  a  little  more  than  half  of  their  planned  on-project  task  hours. 

(1)  Understand  why  the  predicted  project  completion  is  two  years  late? 

(2)  Why  arendteam  members  achieving  planned  on-project  task  hours? 


Before  you  analyze  this  data,  how  do  you  get  this  data? 
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From  having  operationally,  defined  processes  (e.g.,  development  process) 
From  basic,  measurement  data 

-  Operational  measures  (size,  effort,  schedule,  quality) 

-  Measurement  Definitions  (task  hour,  defect,  e  ) 

From  tools 

-  To  record  and  analyze  data 
From  having  a  realistic  plan 

-  developed  by  team  members  who  use  their  own  data  for  estimating 
and  planning 
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Hours 


Planned  and  Actual  Hours  perWeek 


Weeks 


The  team  addressed  the  project  effort  problem. 
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r 


le  Progress  (Earned  Value) 


Earned  Value 


Weeks 


After  initially  falling  farther  behind,  weekly  progress  stabilizes. 
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Report 


TSP  Week  Summary  -  Form  WEEK 


Name 

Consolidation 

Date 

11/8/2007 

Team 

Example  week  8 

Status  for  Week 

"V"  8  Selected  Assembly 

Cycle 

1 

Week  Date 

8/13/2007  SYSTEM 

Task  Hours  %Change 

Baseline 

Current 

%Change 


1280.1 


1358.8 


6.1% 


Weekly  Data 

Schedule  hours  for  this  week 
Schedule  hours  this  cycle  to  dat^ 
Earned  value  for  this  week 
Earned  value  this  cycle  to  dat^ 
To-date  hours  for  tasks  completed 
To-date  average  hours  per  week 
EV  per  completed  task  hour  to  date 


Plan 

47.3 


Project  End  Dates 


2/4/2008 


45.5 


38.3 


1.19 


0.074 


0.092 


The  team  actions  have  been  effective: 

A  Cumulative  hours  have  not  caught  up 
A  The  team  is  9%  ahead  of  schedule 

A  The  predicted  end  date  is  now  2  months  late  rather  than  2  years 
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Build  high  performance  through  teams 

Enable  high  maturity  capabilities  by  building  a  solid  foundation 

CMMI  and  TSP  are  mutually  reinforcingS 

A  CMMI  provides  the  principles  for  process  improvement  and 
organizational  focus 

A  TSP  can  be  useful  for  providing  team  discipline  and  operationalizing 
CMMI  practices 


CMMI®  Current  State  and 

Future  Plans 


Bob  Rassa 

Industry  CMMI  Co-Chair 
Raytheon 

Clyde  Chittister 
Chief  Operating  Officer 
Software  Engineering  Institute 

13  November  2007 
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Topics 


•  Current  Status 

•  Appraisal  Results 

•  Transition  Status 

•  “Beyond  VI  .2”  Workshops 

•  Summary 
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CMMI 


CMMI-vl.2  focused  on  Integrity 

•  Model  improved  to  require  start-up  of  new  projects 
using  appraised  processes 

*  Expanded  appraisal  requirements 

-  Definition  of  appraised  organization 

-  More  disclosure  in  ADS 

-  Relationship  re:  business  objectives  and  HiMat  subprocesses 

-  3-yr  max  period  of  appraisal  validity 

-  SEI  acceptance  of  appraisal  results  prior  to  public  disclosure 

*  High  maturity 

-  Improved  definition  &  understanding 

-  New  training  available 

-  Certified  high  maturity  lead  appraisers 

•  Focus  Topics 


CMMI 


Current  Status  -i 


*  New  policies  announced  and  implemented  for 
version  1.2  appraisals. 

*  Version  1.2  upgrade  training  for  Appraisal  Team 
Members  is  available  online. 

*  CMMI-ACQ  was  released  on  November  1,  2007. 

-  Training 

•  Currently  a  1-day  upgrade  for  those  who  have  previously 
completed  an  Intro,  to  CMMI,  VI. 2  course 

•  Eventually  a  3-day  and/or  an  integrated  course 

-  Appraisals 

•  SCAMPI  A’s  beginning  in  May,  2008 

*  CMMI  adoption  continues  to  grow  worldwide. 
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Current  Status  -2 

*  New  policies  for  version  1 .2  appraisals 

-  Three  year  maximum  period  of  validity 

-  Appraisals  must  be  accepted  by  the  SEI  before 
becoming  public  record 

-  High  maturity  appraisals  must  be  led  by  a  certified  high 
maturity  lead  appraiser 

-  Require  that  level  4/5  appraised  subprocesses  map  to 
organizational  business  objectives 

-  Sampling  rules  and  ADS  revised  and  expanded 

-  Independence  of  lead  appraiser  from  the  business  unit 
being  appraised 

-  CMMI  VI. 1  sunset  date  of  August  31,  2007 

-  Eventual  certification  of  all  lead  appraisers 


5 


CMMI 


Current  Status  -3 


*  Business  Rules  for  CMMI  Focus  Topics 

-  Term  “Extensions”  has  been  changed  to  Focus  Topics 

-  “Extensions”  was  misleading 

-  Some  expectation  that  they  were  appraisable 

-  Promoted  as  accredited  in  extension 

•  Focus  Topics  are: 

-  Intended  to  provide  additional  guidance 

-  Documented  as  Technical  Notes 

-  Reviewed  and  approved  prior  to  release 
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Topics 


*  Current  Status 

*  Appraisal  Results 

*  Transition  Status 

*  “Beyond  VI  .2”  Workshops 

*  Summary 


CMMI 


Appraisal  Synopsis  as  of  6/30/07 

*  Based  on  SCAMPI  vl  .1/vl  .2  Class  A  appraisals 
conducted  since  April  2002  release  through  June 
2007  and  reported  to  the  SEI  by  July  2007. 

*  2,464  appraisals 

*  2,140  organizations 

*  1,417  participating  companies 

•  273  reappraised  organizations 

•  10,338  projects 

•  67.1%  non-USA  organizations 

*  Please  visit  http://www.sei.cmu.edu/appraisal- 
program/profile/pdf/CMMI/2007sepCMMI.pdf,  for  additional 
information  or  to  find  answers  to  questions  you  may  have  about 
this  briefing. 
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cm  mi  Organizational  Type  as  of  6/30/07 

Based  on  Primary  Standard  Industrial  Classification  (SIC)  Code 


Retail  Trade 
0.3% 


Wholesale  Trade 
0.3% 


Mining 

0.1% 


Transportation, 

Communication,  Electric,  Gas 
and  Sanitary  Services 
3.0% 


Business  Services 
34.1% 


Engineering  &  Management 
Services 
20.2% 


Finance,  Insurance  and  Real 
Estate 
6.3% 


Public  Administration 
(Including  Defense) 

6.1% 

Fabricated  Metal  Products 
0.3% 

Primary  Metal  Industries 
0.8% 


Industrial  Machinery  And 
Equipment 
2.0% 


Electronic  &  Other  Electric 
Equipment 
9.5% 


Instruments  And  Related 
Products 
2.8% 


Other  Services 
6.8% 


Transportation  Equipment 
5.1% 


Other  Manufacturing 
Industries 
0.6% 


Based  on  1190  organizations  reporting  SIC  code.  For  more  information  visit: 
http://www.osha.gov/oshstats/sicser.html 
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cm  mi  Organizational  Size  as  of  6/30/07 

Based  on  total  employees  within  area  of  appraised 

organization 

1001  to  2000 

501  to  1000  4A°/o  20001- 


gy  Reporting  Organizational  Types 

as  of  6/30/07 


Military/Government  Agency 


4.4% 


0  200  400  600  800  1000  1200  1400  1600  1800 


Based  on  2131  organizations 


Number  of  Organizations 
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gMMLpjscjpljnes  Selected  for  Appraisals 


For  more  information  about  Allowable  Models  &  Combinations,  visit:  http://www.sei.cmu.edu/cmmi/background/aspec.htmll2 


1000 

950 

900 

850 

800 

750 

700 

650 

600 

550 

500 

450 

400 

350 

300 

250 

200 

150 

100 

50 

0 


•  Maturity  Profile  by  All  Reporting 

Organizations  as  of  6/30/07 


38.1% 


Not  Given  Initial  Managed  Defined  Quantitatively  Optimizing 

Managed 


Based  on  most  recent  appraisal  of  2,140  organizations 
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CMMI.  Countries  Where  Appraisals  Have  Been 
Performed  and  Reported  to  the  SEI  as  of  6/30/07 


Argentina 

Australia 

Austria 

Bahrain 

Belarus 

Belgium 

Brazil 

Bulgaria 

Canada 

Chile 

China 

Colombia 

Costa  Rica 

Czech  Republic 

Denmark 

Dominican  F 

Egypt 

Finland 

France 

Germany 

Hong  Kong 

India 

Indonesia 

Ireland 

Israel 

Italy 

Japan 

Korea,  Republic  Of  Latvia 

Malaysia 

Mauritius 

Mexico 

Morocco 

Netherlands 

New  Zealand 

Pakistan 

Peru 

Philippines 

Poland 

Portugal 

Romania 

Russia 

Singapore 

Slovakia 

South  Africa 

Spain 

Sweden 

Switzerland 

Taiwan 

Vietnam 

Thailand 

Turkey 

United  Kingdom 

Ukraine 

United  Arab  Emirates 

United  States  Uruguay 

Red  country  name:  New  additions  with  this  reporting 
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%  of  Organizations 


CMML  Maturity  Profile  by  All  Reporting  USA 

and  Non-USA  Organizations  as  of  6/30/07 


Not  Given  Initial  Managed  Defined  Quantitatively  Optimizing 

Managed 

15 

Based  on  704  USA  organizations  and  1436  Non-USA  organizations 
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900 

800 

700 

600 
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Number  of  Appraisals  Conducted 

by  Year,  as  of  9/30/07 


JZL 


^  4 


^  #  #  4 


4? 


£ 


N#  ^  ^  ^  rf3  ^  ^  ^ 


□  SPA  ■  CBAIPI(Discontinued  after  12/31/2005)  □  SCAMPI  v.X  QassA 
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Number  of  SCAMPI  VI  .1/1 .2 
Class  A  Appraisals 
Conducted  by  Quarter,  as  of  9/30/07 


CMMI.  Number  of  SCAMPI  V1.1/1.2,  CMMI 


VI. 1/VI .2,  Class  A  Appraisals 
Conducted  by  Year,  as  of  9/30/07 


2002  2003  2004  2005  2006  9/30/2007 


□  SCAMPI  VI .2,  CMMI  V1.2,  ClassA 
■  SCAMPI  V1.1  and/or  CMMI  V1.1,  ClassA 
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Appraisal  Results  Summary 

*  2,464  appraisals  have  been  reported  to  the  SEI  since 
the  release  of  VI. 1  through  June  2007 

-  883  of  these  were  reported  in  the  12  months  from  July  2007  to  June  2008 

*  Commercial/In-house  organizations  report  more  of  the 
appraisals  than  Military/Government  Agency 
organizations 

-  The  highest  percentage  of  Commercial/In-house  organizations  reporting 
appraisals  is  from  outside  the  USA 

-  The  highest  percentage  of  Military/Government  Agency  organizations 
reporting  appraisals  is  from  the  USA 

*  China,  India,  Spain,  France  and  Malaysia  are  reporting 
appraisals  at  particularly  increasing  rapid  rates 

*  CMMI  adoption  continues  to  grow  worldwide. 
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Topics 


*  Current  Status 

*  Appraisal  Results 

*  Transition  Status 

*  “Beyond  VI  .2”  Workshops 

*  Summary 


CMMI 


CMMI  Transition  Status  as  of  9/30/07 


•  Training 

-  Introduction  to  CMMI  -  75,279 

-  Intermediate  CMMI  -  2,612 

•  Authorized 

-  Introduction  to  CMMI  Instructors  -  428 

-  SCAMPI  Lead  Appraisers  -  452 

-  SCAMPI  B&C  Team  Leaders  -  466 

-  SCAMPI  High  Maturity  Lead  Appraisers  -  115 

•  Partners 

-  Introduction  to  CMMI  Training  -  279 

-  SCAMPI  Appraisal  -  285 
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Number  of  Lead  Appraisers 
Authorized  (Cumulative)  as  of  9/30/07 
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Performance  Results  Summary  -i 

as  of  8/30/06 


CMMI 


Improvements 

Median 

#  of  data 
points 

Low 

High 

Cost 

34% 

29 

3% 

87% 

Schedule 

50% 

22 

2% 

95% 

Productivity 

61% 

20 

11% 

329% 

Quality 

48% 

34 

2% 

132% 

Customer 

Satisfaction 

14% 

7 

-4% 

55% 

Return  on 

Investment 

4.0  :  1 

22 

1.7  :  1 

27.7  :  1 

30  Organizations  with  results  expressed  as  changes  over  time 
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CM  ML 

Performance  Results  Summary  -2 

*  For  more  information  on  CMMI  performance  results,  see 

-  An  August  2006  SEI  technical  report  titled  Performance  Results 
of  CMMI-Based  Process  Improvement  (CMU/SEI-2006-TR-004) 

•  It  is  available  on  the  SEI  Web  site  at 
http://www.sei.cmu.edu/publications/documents/06.reports/06tr004. 
html. 

-  The  CMMI  Performance  Results  Web  site  at 
http://www.sei.cmu.edu/cmmi/results.html 

-  The  ROI  site  at  DACS  is  at 
http://www.thedacs.com/databases/roi/ 
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Topics 


*  Current  Status 

*  Appraisal  Results 

*  Transition  Status 

*  “Beyond  VI  .2”  Workshops 

*  Summary 


CM  Ml  CMMI  Beyond-vl.  2  Workshops 


*  Workshops  held  during  CY  2007 

*  Goals  of  Workshops  were  to  promote  free  and 
open  discussion  among  participants  to  identify 
potential  new  methods  to  approach  Process 
Improvement  and  appraisals  thereof  that  can  be 
reflected  in  CMMI  v2.0. 

*  The  ultimate  goal  is  to  evolve  CMMI  to  a  structure 
that  will  better  support  continuous  process 
improvement. 


CMMI 


“Beyond  VI  .2”  Workshops  -i 


Questions  posed  to  Workshop  attendees: 

•  Can  CMMI  be  harmonized  with  other  standards  and 
continuous  process  improvement  efforts? 

•  How  can  we  slim  down  the  CMMI  models  while  still 
preserving  integrity? 

•  Do  we  need  something  different  or  additional  to  define  High 
Maturity? 

•  Should  CMMI  be  used  in  source  selection  and  contract 
monitoring? 

•  Can  repeatability,  coverage  (scope)  and  consistency  of  the 
model  be  improved? 

•  Can  we  identify  next-generation  process  improvement 
methodology?  Are  there  breakthrough  concepts  that  we 
can  apply  to  overall  process  improvement? 

•  Can  we  achieve  one  representation? 

•  Is  the  CMMI  vl.2  Constellation  Strategy  the  right  approach? 

•  How  do  we  make  training  more  efficient  and  effective? 
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“Beyond  VI  .2”  Workshops  .2 


Can  CMMI  be  harmonized  with  other  standards  and 
continuous  process  improvement  efforts? 

•  Agree  that  harmonization  should  be  a  goal,  but  should  not  slow 
progress  too  much 

•  Harmonization  efforts  take  time 

-  (This  may  be  the  only  formal  harmonization  effort) 

Currently, 15288  being  harmonized  with  12207  (ongoing  several 
years).  Recent  work  in  this  area  to  come  out  soon. 

•  Are  there  “standards”  we  want  to  focus  on? 

-  Standards 

-  Process  Improvement  Methods 

-  9001, 14000  (environmental  standard),  AS  9100,  FAA  Standard 
(Aviation  Critical  Safety  Items),  15288, 12207, 15504,  ITIL, 
COBIT,  Sarbanes-Oxley,  632  (Systems  Engineering),  1220, 
Malcolm  Baldridge,  Six  Sigma  not  all  standards  here  are  at  the 
same  level  of  abstraction  ,  PM  BOK  and  OPM3 
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“Beyond  VI  .2”  Workshops  -s 


How  can  we  slim  down  the  CMMI  models  while 
still  preserving  integrity? 

•  How  can  we  make  this  more  user  friendly? 

•  Can  we  slim  down  for  small  projects?  Can  the  model  have  some 
scalability  according  to  various  factors  (e.g.,  project  size,  PoP, 
organization  size)? 

•  How  do  focus  topics  fit  in  with  the  model? 

•  Consider  options  for  packaging  (remove  redundant  stuff  or 
repackage  better) 

•  Consider  fundamental,  intermediate  and  advanced  volumes 

•  Consider  architectural  views  for  appropriate  for  the  different 
using  communities 

•  Consider  streamlining  the  generic  practices  and  look  at  measures 
for  institutionalization 

-  Consider  folding  the  GPs  into  PAs  note  risk  of  losing 
integrity 
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“Beyond  VI  .2”  Workshops  -4 


Do  we  need  something  different  or  additional  to 
define  High  Maturity?  (Part  I) 

•  Focus  on  the  best  practices,  not  focus  on  the  high  maturity 
aspects 

•  Consider  combining  level  4  and  5  into  one  level  because  of  their 
close  tie 

•  4  and  5  are  not  adequately  elaborated  for  implementation  so  these 
may  needs  more  detail  to  drive  proper  behavior 

-  For  example,  if  we  tie  level  4  and  5  to  business  objectives  this 
may  need  to  be  a  practice 

-  If  there  are  additional  requirements  for  the  model  these  can  be 
turned  into  practices.  E.g.,  High  maturity  body  of  knowledge 
and  high  maturity  training 

-  Risk:  adds  more  to  the  model 
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“Beyond  VI  .2”  Workshops  -5 


Do  we  need  something  different  or  additional  to 
define  High  Maturity?  (Part  II) 

•  Consider  redistributing  practices  across  the  levels  to  even  out 
effort  and  expectation 

•  Atlas  study  items  that  impact  the  model  and  results 

•  Consider  maturity  levels  within  PAs  (e.g.,  project  management 
PAs  for  each  level) 

•  Consider  better  interfacing  approaches  with  other  methodologies 
(e.g.,  six  sigma  for  high  maturity) 
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“Beyond  VI  .2”  Workshops  -e 


How  can  repeatability,  coverage  (scope)  and 
consistency  of  the  model  be  improved? 

•  Coverage  (Scope) 

-  Areas  for  consideration: 

•  Operations,  Support,  Transition  to  operations,  Deployment,  Disposal,  Pre¬ 
project,  Proposal,  sustainment,  transition  to  production, 
production/manufacturing,  training 

•  Better  coverage  of  maintenance  and  technical  reviews 

•  Safety,  security,  dependability,  systems  assurance,  environmental 

•  Strategic  planning,  enterprise  management,  finance 

•  Work  force  management  and  development 

•  IR&D,  Advanced  technology,  advanced  technology  test  bed  or  lab 
environment 

•  Small  settings 

•  Product  lines 

•  Business  practices 

•  Information  management  (both  enterprise  and  project) 
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“Beyond  VI  .2”  Workshops  -7 


What  are  the  next-generation  process  improvement 
methodology?  Are  there  breakthrough  concepts 
that  we  can  apply  to  overall  process  improvement? 

•  Consider  how  CMMI  could  interface  with  other  process 
improvement  methodologies  (e.g.  Lean,  six  sigma,  PMBOK, 
theory  of  constraints,  next  generation  IDEAL) 

•  Integration  of  how  people  use  the  various  methodologies 
(same  list  as  above) 

-  Agile  techniques  (extreme  programming),  TSP/PSP 

•  When  everyone  is  level  5,  then  what? 

-  Consider  optimizing  measurements 

•  Consider  an  emphasis  on  process  performance  effectiveness 
and  efficiency,  (e.g.,  effectiveness  6  sigma,  efficiency  LEAN) 

•  How  do  we  measure  program  health? 

-  Need  for  “leading  indicators” 
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“Beyond  VI  .2”  Workshops  -s 

What  representation  should  we  have  (e.g.,  Staged, 

Continuous)?  (Part  I) 

•  Is  the  question  really  level-mania?  (root  cause?) 

-  Level-mania  is  about  doing  the  minimal  work  to  achieve  a  level 
ignoring  what  you  did  to  achieve  the  grade 

-  Levels  are  not  bad,  but  we  need  to  get  the  integrity  of  the  level 
back  up 

•  Provides  a  roadmap  for  projects  to  break  PI  into  pieces 

•  Guidance  for  where  improvement  is  needed 

-  Risk:  has  5  been  around  so  long  that  getting  rid  of  it  will  have 
unintended  consequences? 

•  Maybe  we  have  a  5  level  model  that  only  really  has  3. 

Redesign  the  model  to  address  this 
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“Beyond  VI  .2”  Workshops  .9 


What  representation  should  we  have  (e.g.,  Staged, 
Continuous)?  (Part  II) 

•  How  can  we  make  two  representations  fit  the  same  model? 

•  If  the  model  is  expanded  to  handle  additional  scope,  then  we 
may  need  to  consider  changes  in  the  way  appraisal  results 
are  presented  due  to  sponsor  driven  time  constraints 
packaging 

-  How  do  we  slim  at  the  same  time  as  providing  better 
understanding  and  cover  all  the  things  that  are  need? 

-  As  you  expand  scope,  do  you  need  to  abstract  concepts  versus 
mega  model? 

-  Scope,  slimming  Need  a  vision  and  plan  for  model  evolution 

•  Consider  a  “fixed  size”  approach  and  looking  at  ways  to  present  the 
remaining  information 

•  Consider  pulling  out  OPF  and  coupling  it  with  levels  4  and  5 

•  Don’t  need  to  cover  everything  in  the  same  model 

(packaging)  35 
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“Beyond  VI  .2”  Workshops  -10 


Is  the  CMMI  vl.2  Constellation  Strategy  the  right 
approach? 

*  Alternative  approach:  Start  with  a  CMMI  Model  Framework 
(CMF)  and  add  where  you  need  to  expand  scope  (+  concept) 

•  Are  there  differentiators  for  constellations? 

•  Instead  of  creating  constellations,  encourage  projects  to  do 
what  makes  sense  with  respect  to  what  they  are  doing  using 
the  parent  model 

-  Consider  looking  at  lifecycle  and  what  is  needed  at  each  phase 

-  Can  the  unsophisticated  tailor  the  parent  model  for  their 
perspective? 

*  How  multiple  constellations  can  be  used  in  an  organization 
for  improvement  and  appraisal? 
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“Beyond  VI  .2”  Workshops  -n 


How  can  we  slim  down  the  CMMI  appraisals  while 
still  preserving  integrity?  How  do  we  eliminate  non¬ 
value  add  in  appraisal  and  appraisal  preparation? 
How  do  we  make  appraisals  more  efficient  and 
effective? 

•  Consider  making  the  appraisal  be  focused  on  goals 

•  Add  an  appendix  for  application  practices 

•  Lead  Appraiser  and  the  Appraisal  Team  should  have  enough 
experience  to  review  company  implementations 

•  Make  some  assumptions  that  some  processes  are  in  place 
(e.g.,  assume  project  planning  has  happened,  but  don’t  look  at 
PP  specifically  unless  you  see  something  out  of  place  in  PMC; 
similarly,  could  start  with  IPM  for  a  level  3,  or  QPM  for  a  level  4) 

-  Need  more  guidance  on  where  and  how  you  might  be  able  to  do 
this 

•  More  official  or  formal  mechanisms  for  sampling  coverage 

-  Consider  cost  implications... 
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“Beyond  VI  .2”  Workshops  -12 


Can  we  identify  next-generation  appraisal 
methodology?  Are  there  breakthrough 
concepts  that  we  can  apply  to  overall  appraisals? 

•  Delta  appraisal,  continuous,  incremental,  using  measures 
to  judge  satisfaction,  leading  indicators,  process 
performance  measures,  program  health  (versus  process 
health),  15504  (SPICE),  EIA  732  (percentage  of  practices 
performed,  effectiveness  of  generic  attributes) 

•  Data  reuse  from  previous  appraisals 

•  Appraisal  by  parts 

-  Example,  OSP  separate  from  projects 

-  CMF  separate  from  model  components  unique  to 
constellations 

-  Certify  processes  instead  of  model  (e.g.,  EV  or  SEMP) 

*  Sponsor  commits  to  ongoing  process  improvement 
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“Beyond  VI  .2”  Workshops  -is 


How  do  we  improve  the  trust  and  believability  in  the 
results  of  the  appraisals? 

•  Process  enactment  tools  can  help  with  approval  (workflow 
management) 

•  Need  to  define  consistent  process  for  OSP  tailoring 
approval 

•  Requiring  the  appraisal  be  redone  every  three  years  will 
help  with  believability  (already  in  place) 

-  Consider  notion  of  visits  or  interim  steps  (like  ISO  surveillance 
audits) 

-  Six  month  assessments  focus  on  correlation  between  results 
and  performance  (process  reviews) 

*  Doesn’t  interrupt  the  program  and  not  as  expensive 

-  What  if  you  could  extend  the  life  of  your  appraisal  if  you  did 
interim  review?  What  is  in  it  for  the  organization  to  provide 
incentive  for  the  use  of  reviews? 

•  Would  the  customer  accept  the  results  of  the  interim 
review? 
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“Beyond  VI  .2”  Workshops  -i4 


How  do  we  improve  the  trust  and  believability  in  the 
results  of  the  appraisals? 

•  What  should  the  results  of  a  bad  review  be?  Should  you 
lose  your  level?  Could  you  use  the  delta  appraisal  here? 

•  How  are  organization  changes  that  may  impact  the  process 
capability  tracked? 

-  Consider  adding  a  practice  to  the  model  for  these 
changes 

•  Would  interim  reviews  impact  the  capability  profile 

-  Might  “red  flag”  the  program  or  organization 

•  What  happens  if  you  miss  by  one  practice  in  the  full 
appraisal? 

-  Do  I  get  rewarded  if  I  fix  it?  Consider  delta  approach 

-  Consider  a  more  formal  mechanism  to  track  changes  that  may 
impact  the  process  capability  or  level 
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Next  Steps  for  you... 

*  Send  us  your  ideas 

-  Form  available  on-line  from  SEI 

-  http://www.sei.cmu.edu/cmmi 

-  Submit  like  a  Change  Request 

*  Watch  for  further  opportunities  to  participate... 

-  NDIA  and  SEI  intend  to  conduct  more  Workshops 
in  the  near  future 

-  Announcements  will  be  made  by  NDIA  and  SEI 
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Services  Content  for  CMMI 


•  Draft  CMMI-SVC  started  in  2006 

-  Material  was  sent  to  stakeholder  group  for  review/comment 

-  Work  halted  in  April  2007  to  focus  on  integrity  and  CMMI-ACQ 

•  In  June  2007  issues  on  viability  and  need  for  services 
content  were  posed 

-  Business  case  justification 

-  Usefulness  for  small  businesses 

-  Need  within  Department  of  Defense 

-  Intended  usage 

-  Content  options 

•  Nov  12,  2007  CMMI-SG  discussed  all  issues;  voted  to 
proceed  to  add  Services  content  to  CMMI  Product  Suite 

-  1st  step  is  to  address  all  review  comments  provided  prior  to  work 
halt 

-  2nd  step  is  address  options  for  including  the  content 


CMMI 


Summary 

*  CMMI  has  clearly  reached  far  more  users  than  ever 
envisioned. 

*  Continuing  to  focus  on  appraisal  quality  and 
integrity. 

*  CMMI-ACQ  model  released 

-  2nd  constellation  in  the  CMMI  Product  Suiite 

*  CMMI  “Beyond  vl.2”  Planning  &  Strategy  is 
underway 

*  Services  Content  effort  is  underway 

We  welcome  your  feedback 
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PM  Architecture  Design  as  a 
Critical  Success  Factor  in 
CMMI  Model  Implementation 

November,  2007 


Christen  M.  MacMillan,  PMP 


Implementing  CMMI  into  Your 
Organization 


communications 

Erst&fpn&&  IT  S&lLaErOTiS. 


•  Most  CMMI  efforts  begin  with  noble 
intentions  and  senior  management 
support: 

-Indicates  a  desire  to  improve,  streamline 
and  standardize  how  the  organization 
does  business  and  delivers  quality 


November  2007 
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Implementing  CMMI  into  Your 
Organization 


communications 

Erttfitf  prise  IT  SoluEreriS. 


•  So  why  do  CMMI  initiatives  fail  after 
they’ve  been  authorized  and  resources 
allocated? 

•  What  can  you  do  to  avoid  the  pitfalls? 


November  2007 
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Reasons  Authorized  CMMI 
Efforts  Fail 


E 


communications 

Erst&Tpri&&  ff  So'lulktt& 


1.  Competing  motivators  within  the 
organization 

2.  The  process  implementation  design  was  not 
well  conceived 


November  2007 
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Desire  to  Improve  vs.  Desire  to 
Win  New  Business 


L? 


communications 

Enterprise  IT  SoluEroris. 


Executives 

•  Win  New  Business 

•  Achieve  Rating 

•  Cost  Efficiency 

•  Realists 


Competing  Motivators 

Quality  Organization 

•  Improve  Delivery 

•  Improve  Quality 

•  Mature  Org  Process 

•  Idealists 


*  Executive  sponsorship  for  CMMI  is  often  initially 
more  greatly  influenced  by  obtaining  a  maturity  level, 
rather  than  maturing  the  organization. 

-  Pressure  to  obtain  a  rating 

-  Gain  competitive  advantage 

-  Meet  customer  requirements 

-  Shorter  time  requirements 


November  2007 
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Moving  Beyond  the  “Maturity  L? 
Rating  Motivator” 


Executive  Sponsorship  Continuum 


Motivators 


Signed  authorization 
Some  resources  assigned 


Characterization 


Champion:  fully 
funded  &  resourced 


Lacks 

Champion  & 
Enforcement 


Stakeholders 
held  accountable 
&  actively 
engaged  in  key 
decisions 


November  2007 
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Reasons  Authorized  CMMI 

L* 

Efforts  Fail 

communications 

Erst&Tpn&£  IT  Solutions 

1.  Competing  motivators  exist  within  the 
organization 

2.  The  process  implementation  design  was  not 
well  conceived 


November  2007 
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Reasons  Implementation 
Designs  Fail 


communications 

E prise  IT  SolLaEreriS 


*  Do  not  support  business  goals  or  solve  business 
problems 

*  Do  not  plan  for  managing  organizational  change 

*  Do  not  consider  other  factors  influencing  the  way  the 
organization  does  business 

*  Do  not  factor  other  quality  model  process 
requirements  (i.e.  ISO  registrations,  ANSI  748) 

*  Do  not  account  for  customer  constraints 

*  Do  not  account  for  cost  and  resource  constraints 

*  Provide  no  mechanism  to  lead  the  effort  or  govern 
and  oversee  adherence 

*  The  Process  Design  is  bigger  or  more  complicated 
than  the  organization  needs  or  can  handle 


November  2007 
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Reasons  Implementation 
Designs  Fail  (contd) 


communications 

Erttfif  prise  IT  Sftlulkm& 


•  Do  not  obtain  stakeholder  buy-in  on 
approach,  methods,  and  priorities 

•  Incomplete  business  requirements 

•  The  design  rationale  is  not  fully  planned  and 
communicated 

•  Lack  of  Planning 

•  Lose  sight  of  the  end  goal 


November  2007 
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What  is  the  End  Goal?? 

Ip 

communications 

Erit&Tpn£&  It  Soifcf!km$ 

To  Improve  Project  Performance, 

Delivery,  and  Quality!! 


November  2007 
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Building  Your  Solution:  L? 

Quality  Enterprise  Architecture 


Required  process  rigor  (greater  rigor  =  less  allowable  tailoring) 


November  2007 
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Building  Your  Solution: 

PM  as  a  Foundation  for  CMMI 


communications 

Erstopn&e-  IT  S&lLiEK>ri£ 


PM  practices  against 

*note:  diagram  does  not  represent  complete  mapping 


PM  Practice 

CMMI  PA 

Scope  Definition  & 

Management 

■  RQEM  (L2) 

■  RD  (L3) 

■  VER  (L3) 

■  VAL  (L3) 

Estimation 

■  Cost 

■  LOE  (Level  of  Effort) 

■  Schedule 

■  RQEM  (L2) 

■  RD  (L3) 

■  TS  (L3) 

■  PI  (L3) 

■  MA  (L2) 

■  PP  (L2) 

■  PMC  (L2) 

■  ISM(L3) 

Communication  &  Reporting 

■  GP  2.7/ 2.1 

■  MA  (L2) 

■  PMC  (L2) 

■  IPM  (L3) 

■  RSKM  (L3) 

Knowledge  &  Data 

Management 

■  PM  Repository 

■  CM  System 

■  InfoRQEMation 

Security 

■  CM  (L2) 

■  OPD  (L3) 

■  RD(L3) 

Change  Management 

November  2007 _ 

■  CM  (L2) 

■  PI  (L3) 

■  TS  (L3) 

CMMI  process  areas 


PM  Practice 

CMMI  PA 

Performance  Management 

■  RD  (L3)  ■  PMC  (L2) 

■  Schedule 

■  TS  (L3)  ■  VER  (L3) 

■  Budget 

■  PI  (L3)  ■  VAL  (L3) 

■  Deliverables 

■  IT  (L3)  ■  ISM  (L2) 

Quality  Management 

■  PPQA 

■  Audit  Management  (plan 

■  OPD 

&  schedule) 

■  OPF 

■  Peer  Review 

■  Process  Improvement 

Recommendations/Corre 

ctive  Actions 

Governance 

■  PMC  (L2) 

■  PMO  interface  (if 

■  OPF  (L3) 

applicable) 

■  RSKM  (L3) 

■  Corrective  Action 

■  PPQA  (L2) 

■  MA(L2) 

Resource  Management 

■  OT  (L3) 

■  Staff  management 

■  SAM  (L2) 

■  Asset  management 

■  ISM  (L3) 

■  Subcontractor 

■  IT  (L3) 

Management 

Risk  Management 

■  RSKM  (L3) 

■  PMC  (L2) 

■  PP  (L2) 

November  2007 


12 


Building  Your  Solution: 

L* 

Solution  Steps 

communications 

Erttetf prise  It  S&luErOTiS. 

•  Select  &  Define  Your  PM 
Framework 

•  Develop  Your  PM  Methodology 

•  Establish  a  Governance  System 

•  Ensure  Solution  Meets  Business 
Needs 
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Solution  Step  1 :  Select  & 
Define  Your  PM  Framework 


communications 

E  prise  IT  S&lLaErdilS. 


A  Framework  provides  the  basic  architecture 
for  the  Project  Management  Methodology 

*  Assess  the  organizational  dynamic  and  current  PM  competency 
and  processes 

*  Evaluate  known  frameworks  (i.e.  PMBOK,  home-grown) 

*  Identify  synergies  between  CMMI  PA  requirements  and  other 
quality  best  practices  and  map  to  your  business  needs 

*  Weight  process  attributes  and  level  of  rigor  desired  up  front 

*  Assess  the  characterization  of  your  project  portfolio 

-  Short,  rapid  IT  development  or  long  term  high  risk  combination 
efforts? 

-  Solution  Buyer  or  Solution  Provider? 
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Key  Attributes  of  an  Effective 
PM  Framework 


communications 

Errtfif  prise  IT  S&IliEksiis. 


•  Enables  achievement  of  project  objectives  and  goals 

*  Establishes  foundation  for  monitoring  and  controlling  project 
performance 

-  Identifies  early  performance  indicators 

-  Identifies  performance  shortfalls 

-  Supports  methods  for  corrective  and  preventative  actions 

•  Supports  implementation  of  a  standardized,  but  tailorable 
methodology  that  facilitates  quality  and  timely  development  and 
delivery  of  products  and  services. 

*  Is  flexible  enough  to  integrate  with  other  quality  best  practices, 
models,  and  most  commonly  used  SDLC’s 

-  Build  in  a  way  that  other  quality  models,  frameworks,  and  best 
practices  can  be  “snapped  on”  and  integrated  as  business  needs 
change  and  evolve. 
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Key  Attributes  of  an  Effective 

PM  Framework  (cont  d) 


communications 

Erstepn&e-  IT  SGluErdilS. 


*  Sets  foundation  to  communicate  measures  and  roles 
and  responsibilities 

*  Supports  earlier  stakeholder  and  executive  visibility 
into  performance 

*  Establishes  or  supports  requirements  for  PM 
repository  and  PAL 

*  Identifies  process  interfaces 

*  Defines  PM  process  and  procedural  requirements, 
standards  and  policies  that: 

-  Comply  with  the  CMMI  Model 

-  Comply  with  Business  Requirements 

•  Which  meet  business  objectives 
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Building  Your  Solution: 

L* 

Solution  Steps 

communications 

Erttetf prise  It  S&luErOTiS. 

•  Select  &  Define  Your  PM 
Framework 

•  Develop  Your  PM  Methodology 

•  Establish  a  Governance  System 

•  Ensure  Solution  Meets  Business 
Needs 
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Solution  Step  2:  Develop  Your 
PM  Methodology 


communications 

Erstepn&e-  IT  SGluErdilS. 


A  PM  Methodology  is  the  culmination  and  elaboration  of 

practices  and  methods  by  which  project  management 

is  executed 

•  Elaborate  on  standards  and  requirements  defined  in 
framework/s 

•  Develop  processes,  procedures  and  supporting  documents 

•  Choose  a  specific  PM  Practice  and  follow  the  logical 
progression  of  that  thread 

•  Prioritize  PM  practice  areas  and  implement  in  phases 

•  Begin  with  a  PM  practice  that  helps  solve  immediate  &/or 
significant  problems 
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Building  Your  Solution: 

Ip 

Solution  Steps 

communications 

Erst&Tpn&G-  It  5gIliEh>i1& 

•  Select  &  Define  Your  PM 
Framework 

•  Develop  Your  PM  Methodology 

•  Establish  a  Governance  System 

•  Ensure  Solution  Meets  Business 
Needs 
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Solution  Step  3:  Establish  a 
Governance  System 


communications 

Erttfifpris*  IT  SoluErOfiS. 


A  Governance  System  Should: 

•  Consider  authorizing/creating  a  PMO  to: 

-  Facilitate  identification  of  project  improvement  recommendations 

-  Mentor  “users” 

-  Perform  project/program  audits  to  evaluate  project/program  health  (i.e.  PfM) 

-  Identify  corrective  and  preventative  actions 

•  Establish  Quality  Organization  to 

-  Perform  process  quality  audits 

-  Identify  corrective  and  preventative  actions 

•  Enable  earlier  identification  and  resolution  of  risks 

•  Help  enforce  defined  process  requirements 

•  Establish  requirements  for  process  improvements  &  corrective  actions 

•  Provide  an  independent  escalation  chain  to  executive  management 

•  Facilitate  communication  between  business  operations  &  line  management 
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Building  Your  Solution: 

Ip 

Solution  Steps 

communications 

Erst&Tpn&G-  It  5gIliEh>i1& 

•  Select  &  Define  Your  PM 
Framework 

•  Develop  Your  PM  Methodology 

•  Establish  a  Governance  System 

•  Ensure  Solution  Meets  Business 
Needs 
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Motivators 


Satisfy  customer 
requirements 


Improved  organizational  quality 


id)  id)  id)  id) 


Signed  authorization: 
Some  resources 


Champion:  fully  funded  &  resourced 


Characterization 


•  Continuously  Evaluate  the  Effectiveness  of  the  Implementation 
Approach: 

-  Ensure  defined  goals  are  being  achieved  as  planned 

-  Identify  performance  variances  against  plan  and  take  corrective  action 

-  Ensure  resource  utilization  is  still  appropriate 

-  Let  Process  Improvement  Process  work 
Demonstrate  Business  Value  of  Solution: 

-  Measurably  improve  overall  performance  and  productivity 

-  Standardize  business  processes 

-  Reduce  chaos 

•  Translate  Solution  into  Business  Terms: 

-  Reduce  costs 

-  Increase  the  rate  of  successful  projects  or  business  initiatives 


November  2007 


22 


Ensure  Solution  Meets  Business  L? 

k  I  ^  ^  x ■  x ■  h  I  communications 

Needs  Estimating  Example 


EXAMPLE 


•  The  Value  of  Estimation 
Practices  to  Those  Who  Must 
Implement 

-  Helps  ensure  defendable,  re- 
traceable  estimates  via  a 
standardized  method  and 
documented  BOE  (basis  of 
estimate). 

-  Sets  and  communicates 
stakeholder  expectations, 
system/performance 
boundaries,  requirements 
definition 

-  Facilitates  better  scope 
definition 

-  Defines  criterion  for  change 


•  The  Value  of  Estimation 
Practices  to  Executives 

-  Reduces  financial  and  legal 
risk — particularly  for  FFP 
contracts  or  on  projects  where 
financial  resources  are  limited 

-  Reduces  cost  &  schedule  over¬ 
runs 

-  Increases  win  rate 

-  Produces  more  timely  &  better 
identification  of  requirements 

-  Improves  customer  satisfaction 
through  quality  &  timely  project 
delivery 
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The  Value  of  a  PM  Focus  Up 
Front 


communications 

Erstepn&e-  IT  S&lLaErdilS. 


*  Sets  foundation  for  and  feeds  into  CMMI  process  area 
requirements 

*  Demonstrates  early  value  by  providing  business  leadership, 
sponsors,  project  teams,  with  measurable,  repeatable 
performance  results  before  committing  to  full  cost 

*  Can  be  tailored  at  an  organizational  level  to  accommodate  other 
quality  process  models/best  practices/frameworks:  ITIL;  ISO;  6 
Sigma,  etc. 

*  Fosters  improved  communication  and  defined  roles  & 
responsibilities 

*  Trains  project  teams  to  work  within  a  defined  process 
framework 

-  Realize  value  faster:  motivate  vs.  pull 

*  Facilitates  smoother  management  of  organizational  change 
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The  Value  of  a  PM  Focus  Up 

Front  (cont’d) 


communications 

Erstepn&e-  IT  S&lLaErdilS. 


•  Improves  performance  faster  and  “motivates”  resisting 
stakeholders  to  get  on  board. 

•  Unifies  stove  piped  organizations. 

-  PM  impacts  or  is  impacted  by  business  operations 

•  Establishes  foundation  by  which  your  CMMI  project  can  be 
managed! 

-  Provides  opportunity  to  continuously  improve  your  new  PM 
processes 

•  Key  driver  behind  solution  and  service  success  or  failure 

•  Increases  institutionalization  success 

•  Reduces  risk  of  process  regression  after  a  successful  appraisal 

•  CMMI  requires  other  elements  be  met,  but  you  cannot  meet  any 
CMMI  required  element  without  executing  project  management 
practices 
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War  Stories... 


communications 

Erstopri&&  IT  S&IliEh>i1£ 


...on  the  road  to  a  successful  CMMI  Level  3 
Appraisal 

The  Challenge: 

•  Developing  a  unified  architecture  that  recognized 
many  pre-existing  formal  and  informal  processes 

•  Deciding  how  to  fix  gaps  identified  in  SCAMPI  B 

•  Different  perspectives  on  the  methods,  and  level 
of  process  rigor  needed  to  meet  requirements 
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communications 

Erst&Tpn&fr  IT  Sa'lulktt£ 


War  Stories... 


The  Solution: 

*  Established  a  Process  Action  Team  (PAT)  to  modify 
existing  PM  Process  Framework 

*  The  PAT  included  representation  from  the 
implementing  teams  who  helped  design  the  solution 

*  Leverage  and  build  upon  existing  PM  Processes 
The  Result: 

*  Resolved  most  shortfalls 

*  Achieved  a  Successful  CMMI  Level  3  appraisal!  4' 
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Lessons  Learned 


communications 

Erstopri&&  IT  S&IliEh>i1£ 


•  Develop  your  PM  Framework  &  Methodology  first 

•  Plan  your  CMMI  Implementation  with  the  entire  organizational  process 
architecture  in  mind  when  possible 

•  Don’t  try  to  eat  the  whole  elephant  at  once: 

-  Implement  good  enough  for  now;  improve  process  later 

•  Implement  a  governing  organization  to  oversee  both  Quality  Process 
Adherence  and  Project  Health 

-  Poor  Project  Performance  could  be  an  indicator  that  key  CMMI 
requirements  have  not  been  appropriately  followed 

•  Develop  user-friendly  process  assets  and  repository  (PAL) 

•  Understand  the  dynamics,  structure  and  culture  of  your  organization 

•  Plan  continuous  improvement  activities  to  ensure  your  approach  is 
both  CMMI  compliant  and  meets  business  needs 

•  Be  prepared  for  resistance  and  know  its  source/s 

•  Balance  quick  hits  with  tackling  your  biggest  problem  areas 

•  Communicate,  educate,  listen  &  be  proactive! 
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QUESTIONS? 


communications 

Erst&Tpri&&  IT  SOluEr*i1£ 


Thank  you! 


Christen  MacMillan,  PMP 
Senior  Quality  Process  Manager 
L-3  Communications,  EITS 
Reston,  VA 
703/434-4202  (Office) 
703/798-2852  (Mobile) 
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Proven  Throughout  GM 


>  In  the  1990s,  CMMI  was  adopted  within  GM 

A  Process  and  Capability  Improvement  Framework 
A  Excellent  for  development  organization 

>  There  are  unique  challenges  in  our  acquisition  goals 

AComplexity 

AScalability 

AGIobalization 

>  GM,  the  SEI,  and  Suppliers  partner  to  expand  the  CMMI 

Aldentify  common  organizational  processes 
AModels  for  Acquirer,  Developer,  and  Services 
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and  Project  Management 
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What’s  all  this  ‘churn’  in 
Systems  Engineering 
Standards  and  Models? 


[where  did  they  come  from?e  and  where  are  they  going?] 


CMMI  Technology  Conference 

November  14 , 2007 


Donald  J.  Gantzer 

ODUSD(A&T)  Systems  and  Software  Engineering 
donald.gantzer.  ctr.  osd.mil 

gantzerd(g)svsena-so.  com 

703-412-3668 
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Objectives 


>  To  provide  an  overview  summary  of  key  Systems 
Engineering  [SE]  process  standards  and  models 

>  To  illustrate  a  top  level  comparison  of  them 

>  To  correlate  with  the  Software  Engineering 
Standard 

>  To  indicate  trends  and  usage 

>  To  relate  to  ODUSD(A&T)  System  &  Software 
Engineering  Directotorate  acquisition  Initiatives 

>  To  briefly  address  one  key  process  activity  V 
Technical  Planning  -  as  an  example 


Disclaimer:  The  views  and  opinions  presented  here  are  the 
author’s  and  do  not  necessarily  represent  SAIC  or  DoD  views. 
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Agenda 


>  Introduction 

>  Systems  Engineering  Standards  and  Models 

A  Evolution  of  Standards  &  Models 
A  Summary  of  Standards  &  Models 

-  ISO/IEC  15288:  System  life  Cycle  Processes 

-  AN  SI/E  I A  T  632:  Processes  for  Engineering  a  System 

-  IEEE  1220:  Standard  for  Application  and  Management 

of  the  System  Engineering  Process 

-  CMMI®  -  DEV:  Capability  Maturity  Model  Integrated 

for  Development 

-  DAG/SE;  Defense  Acquisition  Guide/Systems  Engineering 

-  INCOSE  Systems  Engineering  Handbook 

A  A  Mapping  across  standards  and  models 
A  Harmonization  of  ISO/IEC  12207(Standard  for  Information 
Technology  -  Software  Life  Cycle  Processes)  &  ISO/IEC15288 

>  ODUSD(A&T)  Systems  and  Software  Engineering  issues  in  Acquisitions 

>  Summary 

>  Some  Key  References  and  Links 

>  Appendix:  Example  -  Summary  for  Technical  Planning  activities 


Note:  Every  effort  is  made  to  credit  sources  of  material  presented  here 
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items 

ISO/I  EC 
15288 

EIA  -  632 

IEEE  1220 

CMMI®-DEV 

Purpose 

Establish  a 

common 

framework  for 
describing  the 
life  cycle  of 
systemso 

Provide  an 
integrated  set  of 
fundamental 
processes  to  aid  a 
developer  in  the 
engineering  or  re¬ 
engineering  of  a 
systemo 

Defines  the 
requirements  for  an 
enterprise®  total 
technical  effort 
related  to  the 
development  of 
products  and 
processes  that  will 
provide  life  cycle 
support  for  the 
products 

CMMI®)  is  a  process 
improvement  maturity 
model  for  the 
development  of  products 
and  services.  It  consists  of 
best  practices  that 
address  development  and 
maintenance  activities. 

Activities 

25  processes: 

7  Project 

1 1  Technical 

7  Agreement 
and  Enterprise 

33  Requirements  in 

5  groupings  of  13 
processes 

28  requirements: 

14  General 

6  by  Life  Cycle 

Stages 

8  in  SE  Process 

27  process  areas: 
Continuous  model  : 

1 1  Project  Management 

6  Engineering 

5  Support 

5  Process  Management 

Other 
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separate  guide 
for  application] 

-  a  hi-level 
framework 
[descriptive]. 

-120  pgs 
-  in  between  1220 
and  15288  in  scope 
and  details. 
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-85  pgs 
-  less  scope  but 
more  detailed 
[prescriptive]. 

-575  pgs 
-  focus  mainly  on 
development;  much 
supplemental  info. 
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ards  &  Models 


Cycle  Phases 


ISO/IEC  15288 

EIA  -  632 

IEEE  1220 

CMMI®-DEV* 

inferred 

DoD/DAG 
[&  DoDI  5000.2] 

Concept 

Pre-system 

Definition 

Concept 

concept, 

exploration, 

vision 

Concept 

Development 

Development 

System 

Definition, 

Subsystem 

Design, 

Detailed  Design 

System  Def., 
Subsystem  design, 
Detailed  design; 
FAIT 

feasibility,  design, 
development 

Technology 

Development; 

System 

Development; 

Demonstration 

Production 

End  Product, 
Physical 

Integration, 

Test  &  Evaluation 

Production 

production, 

manufacturing, 

delivery 

Production  & 
Deployment: 

LRIP 

Utilization 

Utilization 

operations 

Operations  & 
Support  [O&S]: 
FRIP 

Support 

Support 

support, 

maintenance, 

sustainment 

O&S: 

Sustainment 

Retirement 

Retirement 
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EEE  1220:  SE  Process  -  2005 


Clause  4  -  General  Requirements 

1 .  SE  process 

2.  Polices  &  procedures  for  SE 

3.  Planning  the  technical  effort: 
Prepare/update  engineering  plan; 
schedule;  tech  plans. 

4.  Development  strategies 

5.  Modeling  &  prototyping 

6.  Integrated  repository:  data,  tools. 

7.  Integrated  data  package:  HW,  sw, 
LC  processes,  human. 

8.  Specification  tree 

9.  Drawing  tree 

10.  System  breakdown  structure 

1 1 .  Integration  of  the  SE  effort: 
concurrent  engr.,  Int.  teams. 

12.  Technical  reviews 

13.  Quality  management 

14.  Product  and  process 
improvement:  re-engineering, 
self-assessment,  LL. 

Note:  Standard  includes  detailed  flows 
for  each  activity;  and  an  example 
I —  SEMP  table  of  contents  - 


Clause  6  -  The  SE  Process 


*  Require¬ 
ments/ 
Functional 
/Design 
trade 
studies  & 
assess¬ 
ments 
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1.  Requi 
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irements 

lysis 

1 

f 

i 

i 

◄-> 

2.  Requirements 
Validation 

i 

i 

i 

1 

f 

1 

| 

3.  Functional 
Analysis 

i 

*-►1 

| 

l 

i 

4.  Functional 
Verification 

i 

i 

i 

1 

f 

i 

i 

5.  Synthesis 

1 

*-►1 

i 

y 

i 

i 

6.  Design 
Verification 

i 

i 

i 

i 

i 

7.  Systems 
Analysis* 

1 

r 

8.  Control 
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Plans, 
Directives, 
&  Status 


Acquisition 

Request 


Note:  provides 
detailed  activities 
and  outcomes  for 
each  process 


rocesses  for  Engineering  a  System 

(1999;  reaffirmed  2003) 


(Source:  IN  COSE  SE  Handbook  v2) 


Technical  Management 

\ 

Planning 

Assessment 

Control 

^  Process 

Process 

Process 

/ 

1 


c 

Acquisition  and  Supply 

\ 

Supply  Process 

v 

Acquisition  Process 

J 

CO  NO  PS  &  Requirements 


( 

System  Design 

\ 

Requirements  Definition  Process 

V 

Solution  Definition  Process 

J 

Architectures/Designs 


r 

Product  Realization 

\ 

Implementation  Process 

v 

Transition  to  Use  Process 

J 

Outcomes 

& 

Feedback 


System 

Products 


r 

1 

Products 

r 

Technical  Sunnort 

Systems 

Requirements 

Product 

Product 

Analysis 

Validation 

Verification 

Validation 

^  Process 

Process 

Process 

Process 
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5288:  System  Life-Cycle  Processes 

(2002) 


ENTERPRISE  & 
AGREEMENT 
PROCESSES 


Enterprise  Environment 
Management 


Investment 

Management 


System  Life  Cycle 
Processes 
Management 


Resource 

Management 


Quality 

Management 


Acquisition 


Supply 


Note:  Each  process  has  purpose, 
outcomes,  and  activities 


SYSTEM  LIFE  CYCLE  PROCESSES 
PROJECT  PROCESSES 


Planning 

Assessment 

Control 

Decision-making 


Risk  Management 


Configuration 

Management 


Information 

Management 


TECHNICAL  PROCESSES 


Stakeholder 

Requirements 

Definition 


Requirements 

Analysis 


Architectural  Design 


Implementation 


Integration 


Verification 

Transition 

Validation 

Operation 


Maintenance 


Disposal 


j 


Figurel-1  System  Life  Cycle  Process  Overview  per  ISO/I  EC  15288 
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3  only;  grouped  per  Continuous  model ) 


Category  Process  Area 


Project 

Management 

Project  Planning 

Project  Monitoring  and  Control 

Supplier  Agreement  Management 

Integrated  Project  Management 

Risk  Management 

Support 

Configuration  Management 

Process  and  Product  Quality  Assurance 

Measurement  and  Analysis 

Decision  Analysis  and  Resolution 

Engineering 

Requirements  Management 

Requirements  Development 

Technical  Solution 

Product  Integration 

Verification 

Validation 

Process 

Management 

Organizational  Process  Definition 

Organizational  Process  Focus 

Organizational  Training 

Source:  SEI/CMU 
13 
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Pljnnir  q 


RiS * 


Rcqu:iemenl= 

Managwrwnt 


FnlerFicg 
Man*ganwYl 


C«i6gurit»n- 

Manjctmcn' 


Technn-si  Del-a 

Manjgtmtnt 

Technical 
AswssJitpnl 

Decision 


TECHNICAL  PROCESSES 


Transitljpn 


Requirement 

Prvc-lgpmrnt 


-Logical 

Vcnflcatran 

SflltfliOTi 


Int-agralioh 
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es 

— .  >32 

IEEE  1220 

CMMKD-DEV 

DAG/SE 

15288 

Project 

Planning 

Planning 

Planning  tech  effort; 
Integration  of  SE  effort; 
Development  strategies 

Project  Planning; 
Integrated  PM; 

Product  QA 

Technical  Planning; 
Technical  Data  Mngt. 

Project 

Assessment 

Assessment 

Control 

Measurement  & 
Analysis  [M&A] 

Technical  Assessment 

Project  Control 

Control 

Control;  System 
breakdown  structure 

Project  Monitoring  & 
Control 

Technical  Assessment; 
CM;  Interface  Mngt. 

Decision 

Making 

Systems 

Analysis  (SA) 

Systems  Analysis 

Decision  Analysis  & 
Resolution;  M&A 

Decision  Analysis 

Risk 

Management 

Systems 

Analysis 

Systems  Analysis 

Risk  Management 

Risk  Management 

Configuration 

Management 

(CM) 

CM 

CM; 

Integrated  repository 
and  data  package 

CM; 

Requirements 

Management 

CM; 

Requirements  Mngt.; 
Interface  Mngt. 

Information 

Management 

info 

dissemination 

Integrated  DB/pkg. 

Project  Planning 

Technical  Data 
Management 

Agreement: 
Acquisition  & 
Supply 

Acquisition  & 
Supply 

Supplier  Agreement 
Management  [see  also 

CMMI-ACQ] 

see  other  DAG  chapters 
[e.g.,  Affordability  &  LC 
Resource  Estimates] 

Enterprise: 

Environment, 

Life  Cycle, 
Resource; 

Quality  Mngt. 

Environment  & 
Enterprise 
Support  [e.g., 
resource,  process 
mngt.] 

Quality  Management; 
Product  &  Process 
Improvement 

Process 

Management 

processes; 

Process  &  Product 

QA 

see  other  DAG  Chapters 
[e.g.,  life  cycle  logistics] 
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15288 


Stakeholder 

Requirements 

Definition 


Requirements 

Analysis 


Architectural 

Design 


Implementation 


Integration 


Verification 


Validation 


Transition 


Operation; 
Maintenance;  & 
Disposal 


Your  complimentary 
use  period  has  ended. 

Thank£ucluPS.  ard/Models  Example  Mapping  -  Technical 


632 


Requirements 

Definition 


Systems  Analysis 
(SA) 


Solution  Definition 


Implementation; 

production 


System 

Verification 


Requirements  & 
End  Products 
Validation 


Transition  to  Use 


field  support 


IEEE  1220 


Requirements  Analysis 


Requirements  and 
Functional  Analysis; 
SA;  Modeling 


Functional  Analysis; 
Synthesis;  SA; 
Modeling, 
Specs/drawings 


prototyping; 
fabrication,  assembly, 
production 


Functional  &  Design 
Verification;  Tech 
reviews 


Requirements 
Validation;  Tech 
reviews 


support  stage 


CMMi®-DEV 


Requirements 
Development  & 
Management 


Requirements 

Development 


Technical  Solution 


Technical  Solution 


Product  Integration 


Verification 


Validation 


Product  Integration 


DAG/SE 


Requirements 
Development; 
Logical  Analysis 


Logical  Analysis 


Logical  Analysis; 
Design  Solution 


Implementation 


Integration; 


Verification; 
[+Chap  9  -  IT&E] 


Validation; 
[+Chap  9  -  IT&E] 


Transition 


See  other  DAG 
chapters  [e.g.,  Life 
Cycle  Logistics] 
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>  Following  is  a  quick  overview  of  anticipated  changes  ine 
A  ISO/I  EC  15288 
A  ISO/I  EC  12207 
A  EIA-632 
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>  Why? 

A  Differing  concepts,  structure,  and  audience 

A  First  dalignousing  a  common  nomenclature  structure 
for  ISO/I  EC  15288  &  12207 

A  Later  a  general  life  cycle  process  to  provide  a 
baseline;  focus  on  interoperability  and  integration 

A  Goal  is  a  single  vocabulary,  process  set,  uniform 
architecture,  shared  level  of  prescription,  and  suitable 
across  audiences 


I  >PDF 
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Processes,  Activities,  and  Tasks 


Primary  Life  Cycle  Processes 

Acquisition  Process* 

Supply  Process* 

Development  Process  [to  be  addressed] 

Operation  Process* 

Maintenance  Process* 

Organizational  Life  Cycle 
Processes 

Management  Process** 

Infrastructure  Process* 

Improvement  Process** 

Training  Process** 

*Maps  directly  to  15288:2007 
**  maps  indirectly  to  15288:2007 

19 


Supporting  Life  Cycle  Processes 

Documentation  Process 
Configuration  Management  Process* 
Quality  Assurance  Process** 
Verification  Process* 

Validation  Process* 

Review  process 
Audit  Process 

Problem  Resolution  Process 


Sources:  Anatol  Kark,  Canadian  National  Research 
Center  via  Karen  Richter,  IDA,  in  support  of  DUSD(A&T) 
SSE/SSA;  10/07 
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e  Cycle  Processes 
7 


Agreement 

Processes 


Acquisition  Process 
(Clause  6.1.1) 

Supply  Process 
(Clause  6.1.2) 

Project-Enabling 

Processes 


Life  Cycle  Model 

Management  Process 
(Clause  6.2.1) 


Infrastructure 

Management  Process 
(Clause  6.2.2) 


Project  Portfolio 

Management  Process 
(Clause  6.2.3) 


Human  Resource 

Management  Process 
(Clause  6.2.4) 


Quality  Management 
Process 
(Clause  6.2.5) 


Source:  Anatol  Kark,  Canadian  National  Research 
Center  via  Karen  Richter,  IDA,  in  support  of 
DUSD(A&T)  SSE/SSA;  10/07 


*Changes  are  highlighted 


Project 

Processes 

Project  Planning  Process 
(Clause  6.3.1) 

Project  Assessment  and 
Control  Process 
(Clause  6.3.2) 

Decision  Management 

Process 
(Clause  6.3.3) 

Risk  Management 
Process  (Clause  6.3.4) 

Configuration 
Management  Process 
(Clause  6.3.5) 

Information  Management 
Process 
(Clause  6.3.6) 

Measurement  Process 
(Clause  6.3.7) 

Technical 

Processes 

Stakeholder 
Requirements  Definition 
Process  (Clause  6.4.1) 


Requirements  Analysis 
Process 
(Clause  6.4.2) 


Architectural  Design 
Process 
(Clause  6.4.3) 


Implementation  Process 
(Clause  6.4.4) 


Integration  Process 
(Clause  6.4.5) 


Verification  Process 
(Clause  6.4.6) 


Transition  Process 
(Clause  6.4.7) 


Validation  Process 
(Clause  6.4.8) 


Operation  Process 
(Clause  6.4.9) 


Maintenance  Process 
(Clause  6.4.10) 


Disposal  Process 
(Clause  6.4.11) 
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17: 1 995] 


System  Context  Activities: 

>  System  Requirements  Analysis* 

>  System  Architectural  Design* 


*  Maps  to  I  SO/I  EC  15288:2007 
Technical  processes 


- ► 

>  System  Integration* 

>  System  Qualification  Testing 

>  Software  Installation 

>  Software  Acceptance  Support 

Note:  SW  Reuse  processes  added; 

A  Domain  Engineering 
A  Reuse  Asset  Management 
A  Reuse  Program  Management 


Software  SW  Activities: 

A  SW  Implementation 

A  SW  Requirements 
Analysis 

A  SW  Architecture 
Analysis 

A  SW  Detailed  Design 

A  SW  Coding  &  Testing 

A  SW  Integration 

A  SW  Qualification 
Testing 


Sources:  Anatol  Kark,  Canadian  National  Research  Center  via  Karen  Richter,  IDA,  in  support  of  DUSD(A&T) 

SSE/SSA;  10/07;  James  W.  Moore,  Mitre;  Harmonization  of  Systems  &  Software  Engineering  Processes;  6/07;  brief  to 
ASQ  -DC.  21 
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Iraft  proposal  for  EIA-632A 

f; Source :  GEIA  report;  R.  Harwell,  1 1/05  -  ood?] 

EIA-632  1999 

EIA  -  632A  [date?] 

Planning 

Planning 

Assessment 

Progress  Assessment 

Control 

Control 

Requirements  Definition 

Concept  Definition 

Solution  Definition 

System  Definition 

Product  Realization 
[Implementation/T  ransition] 

System  Realization 

Systems  Analysis 

Mission  &  Systems  Analysis 

Requirements  &  End  products  Validation; 
System  Verification 

System  V&V 

Supply  &  Acquisition 

Customer  &  Supplier  Relationship 
Management 

Enterprise  Support 

Resources  &  Infrastructure 

Enterprise  Support 

Governance 

Enterprise  support 

Life  Cycle  Portfolio  Management 
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A  NDIA-SE  Workshop  on  SE  issues 

A  DoD  SW  Engineering  Workshop  [  via  NDIA-SE] 

A  ODUSD(A&T)  /  SSE  Assessment  &  Support  - 

Program  Support  Reviews  observations 
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Top  5  SE  Issues 

Source:  NDIA  SE  Conference  10/06;  M.  Schaffer  DUSD(A&T)  SSE 


2003  2006 


Lack  of  awareness  of 
SE  importance 

Inconsistent  SE 
practices  across  all  life 
cycle  phases 

Lack  of  adequate 
qualified  resources 

Insufficient  quantity 
and  quality  of  SE 
expertise 

Insufficient  SE  tools 
and  environments 

Inadequate  tools  and 
collaborative 
environments 

Inconsistent 
requirements  definition 

Requirements  not  well 
managed  or  translated 

Poor  initial  program 
formulation 

Insufficient  SE  early  in 
the  life  cycle 
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1. 

2. 

3. 

4. 

5. 

6. 
7. 


The  impact  of  requirements  upon  software  is  not  consistently  quantified 
and  managed  in  development  or  sustainment. 

Fundamental  system  engineering  decisions  are  made  without  full 
participation  of  software  engineering. 

Software  life-cycle  planning  and  management  by  acquirers  and  suppliers  is 

ineffective. 

The  quantity  and  quality  of  software  engineering  expertise  is  insufficient 
to  meet  the  demands  of  government  and  the  defense  industry. 

Traditional  software  verification  techniques  are  costly  and  ineffective  for 
dealing  with  the  scale  and  complexity  of  modern  systems. 

There  is  a  failure  to  assure  correct,  predictable,  safe,  secure  execution 
of  complex  software  in  distributed  environments. 

Inadequate  attention  is  given  to  total  lifecycle  issues  for  COTS/NDI  impacts 
on  lifecycle  cost  and  risk. 


1. 

2. 

3. 

4. 

5. 

6. 
7. 


Source:  NDIA  Top  Software  Issues  Workshop 
August  2006;  K  Baldwin,  DUSD(A&T)  SSE/SSA 
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10  Emerging  Systemic  Issues 

[from  ODUSD(A&T)  SSE/AS  Program  Support  Reviews] 


1.  Management  AlPT  roles,  responsibilities,  authority,  poor  communication 

A  Inexperienced  staff,  lack  of  technical  expertise 

2.  Requirements  A  Creep/stability 

A  Tangible,  measurable,  testable 


3.  Systems  Engineering 

4.  Staffing 

5.  Reliability 

6.  Acquisition  Strategy 

7.  Schedule 

8.  Test  Planning 

9.  Software 


A  Lack  of  a  rigorous  approach,  technical  expertise 
A  Process  compliance 

A  Inadequate  Government  program  office  staff 

A  Ambitious  growth  curves,  unrealistic  requirements 
A  Inadequate  rtest  timeofor  statistical  calculations 

A  Competing  budget  priorities,  schedule-driven 
A  Contracting  issues,  poor  technical  assumptions 

A  Realism,  compression 

A  Breadth,  depth,  resources 

A  Architecture,  design/development  discipline 
A  Staffing/skill  levels,  organizational  competency  (process) 


10.  Maintainability/Logistics  A  Sustainment  costs  not  fully  considered  (short-sighted) 

ASupportability  considerations  traded 

_ _ _ _ Source  DUSD(A&T)  SSE;  M  Schaeffer  8/07 

Major  contributors  to  poor  program  performance 
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2007  -  What  a  Year! 


>  INCOSE  SE  Handbook  v  3.1 

>  Understanding  &  Leveraging  a  Supplier®  CMMI  Efforts;  A  Guidebook  for  Acquirers 

>  CMMI  for  Acquisition  [CMMI-ACQ] 

>  ISO/IEC  15288:2007 

>  ISO/IEC  12207:2007 


...and yet  to  come... 

A  EIA-632? 

A  IEEE-1 220?  [and  ©doptionoof  latest  ISO/IEC  1 5288,  1 2207] 
A  ISO/IEC  24748  -  Life  Cycle  Management  Process  Standard 
A  Further  (Harmonizationoof  ISO/IEC  15288  and12207 
A  CMMI®-DEVv2? 
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ISO/IEC  15288  is  becoming  a  SE  process  ‘reference’  model 
A  IEEE  -  1220;  2005  updated  per  ISO/IEC  15288;  IEEE  cadoptedothe  15288  w 
elaboration;  further  updates  anticipated 
A  CMMI-DEV  vl  .2  uses  SE  standards  and  models  as  sources 
A  ISO/IEC  12207  (SW  Engineering  processes)  is  being  dnarmonizedowith  15288; 

additionally  a  ISO/IEC  24748  Guide  for  LC  Mngt.  in  draft 
A  INCOSE  SE  Handbook  v3.1,  2007  [applies  ISO/IEC  15288;  SE  Certification  will  be 
based  on  it. 

A  Coordination  also  underway  with  the  ISO  9001 

DoD  supported  SE  &  SWE  in  Acquisition  revitalization  activities 
A  DAG /  SE  &  T&E  are  under  revision;  one  area  of  expansion  is  Software 
Engineering 

A  DAU  has  implemented  a  series  of  SE  courses 
A  DoD  Guides: 

-  ‘Integrating  SE  into  DoD  Acquisition  Contracts’ 

-  ‘System  of  Systems  (SoS)  Engineering  ‘Guide  being  piloted 

-  ‘SE  Plan  Preparation  Guide’  revised 

A  NDIA-SE  draft  Systems  Assurance  Guide 


> 
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A&T  Acquisition  and  Technology  [@ODUSD] 

ANSI  i  American  National  Standards  Institute 

DAU  T  Defense  Acquisition  University 

DoD  i  U.S.  Department  of  Defense 

DoDI  i  DoD  Instruction 

EIA  i  Electronic  Industries  Alliance 

GEIAT  Government  Electronics  and  Information  Technology  Association 

IECi  International  Electrotechnical  Commission 

IEEE  i  Institute  for  Electrical  and  Electronics  Engineers 

INCOSE  i  International  Council  on  Systems  Engineering 

ISO  T  International  Standards  Organization 

ITT  Information  Technology 

NDIAT  National  Defense  Industries  Association  [SE  division] 

PM  I  i  Project  Management  Institute 
SE  T  Systems  Engineering 

SEI T  Software  Engineering  Institute  [@Carnegie  Mellon  U.] 

SEMP  i  SE  Management  Plan 

SEP  \  Systems  Engineering  Plan 

SSCI T  Systems  and  Software  Consortium 

SSA  \  Software  Engineering  and  Systems  Assurance 

SSE  V  Systems  &  Software  Engineering  Directorate  [ODUSD  (A&T] 

SWE  V  Software  [SW]  Engineering 


> 

> 

> 

> 

> 

> 

> 

> 

> 

> 

> 

> 

> 

> 

> 

> 

> 

> 

> 

> 

> 
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ISO/IEC  15288:  2002  System  Engineering  T  System  Life  Cycle 
Processes  [new  version  released  2007] 

EIA/IS  -  632: 1998  -  Processes  for  Engineering  a  System 
IEEE  1220:  2005  Application  and  Management  of  the  Systems 
Engineering  Process 

CMMI®-DEVT  Capability  Maturity  Model  Integration®  for 
Development  vl  .2  (2006)  [updating  underway] 

Defense  Acquisition  Guide,  Chapter  4  -  Systems 
Engineering;  Defense  Acquisition  University,  2004  [being 
updated] 

Understanding  and  Leveraging  a  Supplier’s  CMMI  Efforts; 

DUSD(A&T)  SSE;  2007 

CMMI®  -  ACQ:  Adapting  CMMI®  for  Acquisition 
Organizations:  A  Preliminary  Report;  2006  [new  model  report 
released  1 1/07  by  SEI/CMU] 

INCOSE  Systems  Engineering  Handbook,  v3.1;  8/2007 

PMBOK®  PMI’s  Project  Management  Book  of  Knowledge 
IEEE/EIA  12207  [adopted  ISO/IEC  12207];  1997  [new  version 
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released  2007] 
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and  Links 


>  nSE  Standards  &  Models  Comparedq  J.  Lake  (SMi)  and  S.  Sheard  (SPC),  INCOSE  2004 

>  n Evolution  of  a  Standard  EIA-632q  R.  Harwell,  INCOSE  2006 

>  n Special  Feature:  Standards  in  Systems  Engineering^  INCOSE  Insight ;  April  2007  (see  particularly  K.  Crowder,  D. 
Kitterman,  T.  Doran,  R.  Harwell,  and  S.  Arnold  articles) 

>  CMMI  -  Next  Steps;  Kristen  Baldwin,  ODUSD(A&T)  SSE/SSA;  CMMI  technology  Conference;  November,  2007 

>  r Harmonization  of  Systems  and  Software  Engineering  Processesq  James  W.  Moore;  Mitre;  June,  2007,  brief  for 
ASQ-DC  meeting 

>  Issue  on  Systems  Engineering]  CROSSTALK,  STSC;  October  2007 

Links: 

>  ANSI/EIA-632:  http://www.qeia.orq/index.asp?bid=552 

>  CMMI:  http://www.sei.cmu.edu/cmmi/ 

>  DAU-DAG:  http://akss.dau.mil/daq/ 

>  IEEE -1220:  http://www.techstreet.com/cqi-bin/detail7product  id= 1260785 

>  IEEE  Standards:  http://www.ieee.org/web/standards/home/index.html 

>  INCOSE  T  Standards  site:  http://www.incose.org/practice/techactivities/standards.aspx 

>  INCOSE  Guide  to  SE  BoK:  http://g2sebok.incose.org/ 

>  ISO:  http://www.iso.org/iso/home.htm 

>  ISO/IEC  15288:  http://www.1 5288.com/ 

>  NDIA-SE:  http://www.ndia.orq/Template.cfm?Section=Divisions  [then  select  SE] 

>  ODUSD  (A&T)  SSE:  http://www.acq.osd.mil/sse/ 

>  Systems  &  Software  Consortium:  http://www.systemsandsoftware.org/ 

Note:  If  you  have  problems  locating  references,  contact  me  at  gantzerd@saic.com 
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Radices  to  Technical  Planning 


It  was  found  very  difficult  to  ‘map’  planning  activities  from  the 
various  standards  &  models  at  this  level  of  detail  -  so  decision 
was  made  to  just  summarize  each  for  your  own  consideration 

However,  it  is  concluded  that  some  very  basic  activities  that 
need  to  be  accomplished  for  planning  are . . . 

-  the  what,  why,  who,  when  and  how! 


ISO/IEC  15288 
EIA  -  632 
IEEE  1220 
CMMI®-DEV 
PM  BoK 

INCOSE  SE  Handbook 

ODUSD(A&T)  SSE  Technical  Planning  considerations 
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5288  -  Project  Planning  Activities 


Purpose:  to  produce  and  communicate  effective  and  workable  project  plans 

>  Identify  the  project  objectives  and  constraints 

>  Define  the  project  scope  as  established  in  the  agreement 

>  Establish  a  WBS  based  on  evolving  system  architecture 

>  Define  and  maintain  a  project  schedule  based  on  project  objectives  and  work 
estimates 

>  Project  achievement  criteria  for  the  life  cycle  stage  decision  gates,  delivery  dates 
and  major  dependencies  on  external  inputs  or  outputs 

>  Define  the  project  costs  and  plan  a  budget 

>  Establish  the  structure  of  authorities  and  responsibilities  for  project  work 

>  Define  the  infrastructure  and  services  required  by  the  project 

>  Plan  the  acquisition  of  materials,  goods  and  enabling  system  services  supplied 
from  outside  the  project 

>  Generate  and  communicate  a  plan  for  technical  mgmt.  of  the  project,  including  the 

reviews 

>  Define  the  project  measures  to  be  generated  and  the  associated  data  to  be 
collected,  validated  and  analyzed 

>  Generate  a  project  quality  plan 
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2  -  Technical  Planning 


>  Process  Implementation  Strategy 

A  stakeholders,  applicable  docs,  process  approaches,  LC 
phases,  integration,  reporting  requirements, 
implementation 

>  Technical  Effort  Definition 

A  Requirement  types,  db,  risk  mngt.  process  metrics, 
metrics/quality,  cost  objectives,  TPMs,  tasks,  methods 
&  tools,  technology 

>  Schedule  &  Organization 

A  Event&  calendar  based  schedules,  resources, 
staffing/disciplines,  team /  org  structure 

>  Technical  Plans 

A  Engineering,  Risk  mngt.,  Tech  Review,  V  &V,  other 

>  Work  Directives 

A  Work  packages,  work  authorizations 

34  Source:  EIA-632 

SE  standards  &  models 


I  >PDF 

*  Complete 


Your  complimentary 
use  period  has  ended. 
Thank  you  for  using 
PDF  Complete. 


and  Expanded  Features 


djGantzer 


0  -  Planning  the  Technical  Effort 


“Prepare  and  Implement  the  technical  plans  and  schedules  to  guide 
the  project  toward  accomplishment  of  its  objectives  and  proper 
conclusion.  ” 

A  Engineering  Plan  [example  SEMP  content] 

A  Master  and  Detail  Schedules 
A  Technical  Plans 
A  Developmental  Strategies 
A  Modeling  &  Prototyping 

A  Integrated  Repository,  Data,  Tools,  and  Integrated  Data  Package 
A  Hw,  SW,  Humans 
A  Life  Cycle  Processes 
A  Specifications  and  Drawing  Trees;  SBS 
A  Integration  the  SE  Effort 
A  Tech  Reviews 
A  Quality  Management 
A  Product  &  Process  Improvement 
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-  DEV  -  Project  Planning 


Purpose:  to  supply  and  maintain  plans  that  define  project  activities. 

>  Establish  Estimates 

A  Estimate  scope 

A  Establish  Estimates  of  work  products/attributes 
A  Define  life  cycle 
A  Determine  effort  &  cost  estimates 

>  Develop  Project  Plan 

A  Establish  budget  &  schedule 
A  Identify  risks 
A  Plan  for  data  management, 

A  Plan  for  resources;  Needed  knowledge  &  skills 
A  Plan  stakeholder  involvement 
A  Establish  the  Plan 

>  Obtain  commitment  to  the  Plan 

A  Review  plans  that  affect  project 

A  Reconcile  work  &  resource  levels  U  Q^er  ^ey  process  area  relationships:  -  Requirements 
A  Obtain  commitment  Development,  Project  Monitoring  &  Control,  Supplier 

Agreement  Mngt. ,  Integrated  PM,  Risk  Mngt., 
Measurement  &  Analysis,  . . . 
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,m.  Generic  Practices  for  all  process  areas 


>  Perform  the  planning  process 

>  Establish  &  maintain  an  Org  policy  for  planning 
process 

>  Plan  the  planning  process 

>  Provide  resources 

>  Assign  responsibility 

>  Train  people 

>  Manage  configurations 

>  Identify  and  involve  relevant  stakeholders 

>  Monitor  and  control  the  planning  process 

>  Objectively  evaluate  adherence  to  the  planning 
process 

>  Review  status  with  higher  level  management 
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SE  Handbook  -  Planning  Process 


Figure  5-2  Context  Diagram  for  the  Project  Planning  Process 


Source:  INCOSE  SE  Handbook  v3. 1 
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agement  -  Book  of  Knowledge  (PMBOK) 

agement  area) 


>  Scope* 

>  Integration 

(charter,  scope  statement, 

PMP) 

>  Communication* 

>  Risk* 

>  Quality* 

>  Human  Resources* 

>  Time  (definition, 

Sequencing,  estimation)  *  Note:  DoD  PMBoK  Extension  (2003  also  covers 

>  Cost  (estimation,  SE,  SW Acquisition,  Logistics,  T&E,  Manufacturing 

budgeting) 

>  Procurement  (purchase, 
acquisition,  contracting) 

*  Apply  Planning ,  Execution  & 

Control  to  each  area 

Source:  www.PMI.org ;  3rd  Edition,  2004 
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>  Manage  a  Comprehensive  Set  of  Requirements 

A  Define  project  scope  w  key  stakeholders  [FoS,  SoS] 

A  Formulate,  assess,  select  the  preferred  system  concept 
A  Develop  explicit  and  testable  system/project  requirements 
A  Develop  a  WBS  [products  &  process] 

>  Resource  &  Staffing  to  the  Technical  Plan 

A  Organize  and  staff  the  project  team  [  PM,  Lead  SE,  IPTs] 

A  Estimate  the  time  and  resource  requirements  [IMS,  EVMS 
A  Develop  a  project  critical  path 
A  Develop  a  project  budget 

>  Develop  and  Managing  Technical  Baselines 

A  Identify,  manage,  and  mitigate  project  risks  [technical] 

A  Manage  project  changes  and  customer  expectations 

>  Managing  Event-based  Technical  Reviews 

>  Integrating  Tech  Planning  into  overall  Program  Planning  & 
Management  Context  [IMP/IMS,  EVMS,  program  Risks] 


Note:  DoD  is  updating  DAG/S E,  DoDI 
5000.2,  and  SEP  Prep  Guide  just  updated 

djGantzer 
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)  Systems  and  Software 
/ing  Missi on  Statement 


>  Shape  acquisition  solutions  and  promote  early  technical  planning 

>  Promote  the  application  of  sound  systems  and  software  engineering, 
developmental  test  and  evaluation,  and  related  technical  disciplines 
across  the  Department's  acquisition  community  and  programs 

>  Raise  awareness  of  the  importance  of  effective  systems  engineering 
and  drive  the  state-of-the-practice  into  program  planning  and 
execution 


>  Establish  policy,  guidance,  best  practices,  education,  and  training  in 
collaboration  with  academia,  industry,  and  government  communities 


> 


Provide  technical  insight  to  program  managers  and  leadership  to 
support  decision  making 

Source:  DOD(A&T)  SSE;  M  Schaeffer,  8/07 
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Background 


Software  quality  is  governed  largely  by  the  quality  of  the 
processes  involved  in  developing  or  acquiring,  and  maintaining 
it. 

Carnegie  Mellon  UniversityQ  Software  Engineering  Institute 
(SEI)  has  developed  models  and  methods  that  define  and 
determine  organizationsOsoftware  process  maturity. 

These  provide  a  logical  framework  for  baselining  an 
organization^  current  process  capabilities  (i.e.,  strengths  and 
weaknesses)  and  providing  a  structured  plan  for  incremental 
process  improvement. 
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CMMI  at  GAO 


A  Lead  Appraiser 

A  Approximately  150  individuals  trained  in  various  models 
primarily  CMMI  and  others  over  the  years  (SW-CMM, 
SA-CMM,  and  P-CMM) 
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CMMI ACQ  Users 


A  Organizations  that  acquire  products  or  services  that  include 
software 

Organizations  that  contract  for  systems  solutions  which 
contain  software 

Integrated  product  teams  assisting  in  acquisition 
management 
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Aa 


gencies  are 


Increasingly  contracting  out  significant  portions  of  IT 
development. 

CPartneringOwith  contractors 

Under  the  impression  that  the  contractor®  processes  are 
meet  their  needs 

Letting  contractors  manage  themselves 
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doting  Requires  Discipline 


A  Federal  agencies  assume  contracting 


absolves  them  of  acquisition-related  responsibilities 

A  e.g.  no  need  to  define  requirements,  needs  minimal  tracking 
and  oversight, ... 

alleviates  them  from  risk 

addresses  development  immaturity 

coupled  with  COTS  produces  immediate  results 
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uisition  Misconceptions 


A  Use  commercial  methodology  to  guide 
systems  prioritization 

A  Partnering  with  Prime  to  jointly  dbuildO 

A  Transferring  all  risk  to  the  contractor 
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aisition  Issues  Discovered 


Aa 


gencies 


Do  not  grasp  difference  between  acquisition  and 
development 

Believe  that  by  hiring  contractors  they  are  acquiring  a 
system 

Let  contractors  define  requirements 

Assume  a  lifecycle  methodology  solves  all  problems 
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T  Do  not  fully  understand  what  they  QvantO 
l  Unable  to  articulate  what  they  do  QvantO 
i  Do  not  have  a  QoadmapO 
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'ACQ  Facilitates  Discipline 


A  The  CMMI-ACQ 


T  Provides  the  dbuyerOa  roadmap  for  acquisition  process 
improvement 

I  Enables  the  dbuyerOto  measure  and  improve  selected 
areas 

I  Facilitates  management  and  control  of  the  acquisition 
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A  GAO  audits  assess  every  specific  and 
generic  practice  for  Class  A. 

A  GAO  has  also  conducted  various  Class  B 
and  C  appraisals. 

A  GAO  works  with  the  audited  agency  to 
identify  specific  process  areas  that  are 
applicable  and  may  pull  from  multiple 
models  (CMMI-DEV  and  CMMI-ACQ) 
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GAO  Methodology 


A  Details  findings  provided  by  GAO  audits 


assist  in  process  improvement 
provide  a  way  to  prioritize  improvement 
I  enable  building  on  areas  of  strength 
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Conclusion 


A  Contracting  out  is  not  a  panacea  for  lack  of 
acquisition  processes 

A  GAOCS  use  of  CMMI-ACQ  pinpoints 
specific  areas  for  improvement 
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Country 

Number  of 
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Maturity 
Level  1 
Reported 

Maturity 
Level  2 
Reported 

Maturity 
Level  3 
Reported 

Maturity 
Level  4 
Reported 

Maturity 

Level  5 

Reported  Country 

Number  of 
Appraisals 

Maturity 
Level  1 
Reported 

Maturity 
Level  2 
Reported 

Maturity 
Level  3 
Reported 

Maturity 
Level  4 
Reported 

Maturity 
Level  5 
Reported 

Argentina 

26 

No 

Yes 

Yes 

Yes 

Yes 

Malaysia 

29 

No 

Yes 

Yes 

No 

Yes 

Australia 

26 

Yes 

Yes 

Yes 

Yes 

Yes 

Mauritius 

10  or  fewer 

Austria 

10  or  fewer 

Mexico 

29 

Yes 

Yes 

Yes 

Yes 

Yes 

Bahrain 

10  or  fewer 

Morocco 

10  or  fewer 

Belarus 

10  or  fewer 

Netherlands 

10  or  fewer 

Belgium 

10  or  fewer 

New  Zealand 

10  or  fewer 

Brazil 

58 

No 

Yes 

Yes 

Yes 

Yes 

Pakistan 

10  or  fewer 

Bulgaria 

10  or  fewer 

Peru 

10  or  fewer 

Canada 

38 

No 

Yes 

Yes 

Yes 

Yes 

Philippines 

17 

No 

Yes 

Yes 

No 

Yes 

Chile 

17 

No 

Yes 

Yes 

No 

Yes 

Poland 

10  or  fewer 

China 

321 

Yes 

Yes 

Yes 

Yes 

Yes 

Portugal 

1 0  or  fewer 

Colombia 

16 

No 

Yes 

Yes 

Yes 

Yes 

Romania 

1 0  or  fewer 

Costa  Rica 

1 0  or  fewer 

Russia 

1 0  or  fewer 

Czech  Republic 

1 0  or  fewer 

Singapore 

10 

Denmark 

1 0  or  fewer 

Slovakia 

1 0  or  fewer 

Dominican  Republic  10  or  fewer 

South  Africa 

10  or  fewer 

Egypt 

25 

No 

Yes 

Yes 

Yes 

Yes 

Spain 

55 

No 

Yes 

Yes 

Yes 

Yes 

Finland 

1 0  or  fewer 

Sweden 

1 0  or  fewer 

France 

94 

Yes 

Yes 

Yes 

Yes 

Yes 

Switzerland 

1 0  or  fewer 

Germany 

41 

Yes 

Yes 

Yes 

Yes 

Yes 

Taiwan 

71 

No 

Yes 

Yes 

No 

Yes 

Hong  Kong 

10 

Thailand 

1 0  or  fewer 

India 

256 

No 

Yes 

Yes 

Yes 

Yes 

Turkey 

1 0  or  fewer 

Indonesia 

1 0  or  fewer 

Ukraine 

1 0  or  fewer 

Ireland 

1 0  or  fewer 

United  Arab  Emil  1 0  or  fewer 

Israel 

12 

No 

Yes 

Yes 

No 

Yes 

United  Kingdom  57 

Yes 

Yes 

Yes 

Yes 

Yes 

Italy 

12 

No 

Yes 

Yes 

No 

No 

United  States 

859 

Yes 

Yes 

Yes 

Yes 

Yes 

Japan 

197 

Yes 

Yes 

Yes 

Yes 

Yes 

Uruguay 

1 0  or  fewer 

Korea,  Republic  Of 

87 

Yes 

Yes 

Yes 

Yes 

Yes 

Viet  Nam 

1 0  or  fewer 

Latvia 

1 0  or  fewer 
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The  possible  options  for  assessment  and  surveillance 
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The  Addison-Wesley  SEI  Series  Book  and 
A  A  Guide  to  the  CMMI 
A  CMMI:  A  Frameworke 
A  CMMI  Assessments 
A  CMMI  Distilled:  Second  Edition 
A  CMMI  SCAMPI  Distilled 
A  CMMI  Survival  Guide 
A  CMMI:  Un  Itineraire  Fleche 
A  De  kleine  CMMI 
A  Interpreting  the  CMMI 
A  Making  Process  Improvement  Work 
A  Practical  Insight  into  CMMI 
A  Real  Process  Improvement  Using  the  CMMI 
A  Systematic  Process  Improvement  Using  ISO  9001:2000  and  CMMI 
A  Balancing  Agility  and  Discipline 
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A  Using  CMMI-DEV  for  sourcing 
A  Interpreting  CMMI: 

6  for  Operational  Organizations 
6  for  COTS  Based  Systems 
6  for  Service  Organizations 
6  for  Marketing 
A  Using  CMMI  with: 

6  TS P/PSP 

6  Earned  Value  Management 
6  Product  Line  Practices 
6  Lean  Six  Sigma 

A  Supplementing  CMMI  for  Safety  Critical  Development 

A  Demonstrating  the  Impact  and  Benefits  of  CMMI  (and  Web  pages  i 
www.sei.cmu.edu/cmmi/results) 
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Improvements 

Median 

#  of  data 
points 

Low 

High 

Cost 

34% 

29 

3% 

87% 

Schedule 

50% 

22 

2% 

95% 

Productivity 

61% 

20 

11% 

329% 

Quality 

48% 

34 

2% 

1 32% 

Customer  Satisfaction 

14% 

7 

-4% 

55% 

Return  on  Investment 

4.0  :  1 

22 

1.7  :  1 

27.7  :  1 

A  N  =  30,  as  of  August  2006 

A  Organizations  with  results  expressed  as  change  over  time 
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A  Over  75,000  people  have  been  trained 
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Now  we  want  to  continuously  improvee 
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Reduce  complexity  &  size 
Increase  coverage 

Increase  confidence  in  appraisal  results 
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Eliminated  the  concepts  of  advanced  practices  and  common  features 
Incorporated  ISM  into  SAM;  eliminated  Supplier  Sourcing  (SS)  addition 
Consolidated  and  simplified  the  IPPD  material 
All  definitions  consolidated  in  the  glossary 

Adopted  a  single  book  approach  (i.e.,  will  no  longer  provide  separate 
development  models) 

Report  size  reduced  15%  from  either  predecessor;  PAs  reduced  12% 
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Added  hardware  amplifications 

Added  two  work  environment  practices  (i.e.,  one  in  OPD  and  one  in  IPM) 

Added  goal  and  two  practices  in  OPF  to  emphasize  importance  of  project 
startup 

Updated  notes  (including  examples)  where  appropriate  so  that  they  also 
address  service  development  and  acquisition  of  critical  elements 

Updated  name  to  CMMI  for  Development  (CMMI-DEV)  to  reflect  the 
expanded  coverage 
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Improved  the  Overview  section  (Part  One) 

Improved  clarity  of  how  GPs  are  used 

A  Moved  generic  goals  and  practices  to  Part  Two 

A  Added  explanation  of  how  process  areas  support  the  implementation  of 


GPs 


A  Added  GP  elaborations  for  GP  3.2 

Improved  the  glossary  (e.g.,  higher  level  management,  bidirectional 
traceability,  subprocess) 

Limited  the  process  areas  that  can  be  considered  mot  applicableoto  SAM. 

Clarified  material  throughout  the  model  based  on  over  1000  change 
requests 
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iuct  and  Process  Development  cmml 


IPPD  material  is  being  revised  significantly. 

A  Organization  Environment  for  Integration  PA  removed  and  material  moved 
to  Organizational  Process  Definition  (OPD)  PA. 

A  Integrated  Teaming  PA  removed  and  material  moved  to  Integrated  Project 
Management  (IPM)  PA. 

A  IPPD  goals  have  been  consolidated. 

6  rEnable  IPPD  Managementoin  OPD 

5  (Apply  IPPD  Principlesoin  IPM 

A  Overall  material  condensed  and  revised  to  be  more  consistent  with  other 
PAs. 
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Sourcing 


SW 
Related 
Examples 


Integrated  Product  and 
Process  Development 
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SE 

Related 

Examples 


CMMI  Core 


VI  .2 


IPPD 


4. 
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Organizational  Goal 
(OPD) 

Project  Goal  (IPM) 


SW 
Related 
Examples 


SE 
Related 
Examples 


HW 

Related 
Examples 
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Specific  Goal 

Specific  Practice 

Establish  Supplier 
Agreements 

1.1  -  Determine  Acquisition  Type 

1.2  -  Select  Suppliers 

1.3  -  Establish  Supplier  Agreements 

Satisfy  Supplier 
Agreements 

2.1  -  Execute  the  Supplier  Agreement 

2.2  -  Monitor  Selected  Supplier 

Processes 

2.3  -  Evaluate  Selected  Supplier  Work 

Products 

2.4  -  Accept  the  Acquired  Product 

2.5  -  Transition  Products 

VI. 1  SP2.1  rReview  COTS  Products, owas  eliminated.  (Identify 
candidate  COTS  products  that  satisfy  requirementsois  a  new 
subpractice  under  the  Technical  Solutions  Process  Area  SP1.1, 
(Develop  Alternative  Solutions  and  Selection  Criteria. 6 
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VI. 1 


VI  .2 


SG  1  -  Determine  Process  Improvement 
Opportunities 

1.1  -  Establish  Organizational  Process  Needs 

1.2  -  Appraise  the  Organization’s  Processes 

1.3  -  Identify  the  Organization’s  Process 
Improvements 

SG  2  -  Plan  and  Implement  Process 
Improvement  Activities 

2.1  -  Establish  Process  Action  Plans 

2.2  -  Implement  Process  Action  Plans 

2.3  -  Deploy  Organizational  Process  Assets 

2.4  -  Incorporate  Process-Related  Experiences 
into  the  Organizational  Process  Assets 


SG  1  -  Determine  Process  Improvement 
Opportunities 

1.1  -  Establish  Organizational  Process  Needs 

1.2  -  Appraise  the  Organization’s  Processes 

1.3  -  Identify  the  Organization’s  Process 
Improvements 

SG  2  -  Plan  and  Implement  Process  Improvement 

2.1  -  Establish  Process  Action  Plans  I  New 

2.2  -  Implement  Process  Action  Plans 

SG  3  -  Deploy  Organizational  Process  Assets  and 
Incorporate  Lessons  Learned 

3.1  -  Deploy  Organizational  Process  Assets 

3.2  -  Deploy  Standard  Processes 

3.3  -  Monitor  Implementation 

3.4  -  Incorporate  Process  Related  Experiences 
into  the  Organizational  Process  Assets 
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Method  implementation  clarifications 

A  interviews  in  mrtualdorganizations 

A  practice  characterization  rules 

A  organizational  unit  sampling  options 

Appraisal  Disclosure  Statement  (ADS)  improvements 

A  reduce  redundancy  with  other  appraisal  documents 

A  improve  usability  for  sponsor  and  government 

A  Level  4,5  mapping  to  business  objectives 

A  require  sponsor®  signature  on  the  ADS 

A  require  all  team  members  to  show  agreement  on  findings 

A  Both  VI  ,1  and  VI  .2  ADS  reflect  these  today 

Appraisal  team  will  have  responsibility  for  determination  of  napplicabilityofor  SAM 

Maturity  level  and  capability  level  validity  period  T  3  years,  given  1  year  of  VI  .2 
availability 
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CcirnmgirMrlliwi 
Software  Engineering  Institute 


List  of  Published  SCAMPI  Appraisal  Results 


ORGANIZATION  NAME: 

SPONSOR  NAME: 

LEAD  APPRAISER  NAME: 

SEI  PARTNER: 

APPRAISAL  END  DATE: 

MATURITY  LEVEL  ASSIGNED: 
APPRAISED  ORGANIZATIONAL  UNIT: 


Satyarn  Computer  Services  Ltd. 

Nagaraj  Chevour 

Raghavan  Nandyal 

SITARA  Technologies  Pvt.  Ltd. 

4/3/2004 

5 


Entity  Name:  SRU  GE-GDC 

Location(s):  Secunderabad,  AP,  India 

CMMI  MODEL  USED:  CMMI-SW/IPPD,  VI  .1 ,  Continuous 

APPRAISAL  METHOD  USED:  SCAMPI  vl  .1 


MODEL  SCOPE  &  CAPABILITY  RATINGS  ASSIGNED: 


Process  Management 

Project  Management 

Engineering 

Support 

OPF  Capability  Level  3 

PP  Capability  Level  4 

REQM  Capability  Level  3 

CM  Capability  Level  3 

OPD  Capability  Level  3 

PMC  Capability  Level  4 

RD  Capability  Level  4 

PPQA  Capability  Level  3 

OT  Capability  Level  3 

SAM  Not  Applicable 

TS  Capability  Level  5 

MA  Capability  Level  3 

OPP  Capability  Level  3 

IPM  Capability  Level  3 

PI  Capability  Level  3 

DAR  Capability  Level  3 

Ol D  Capability  Level  3 

RSKM  Capability  Level  4 

VER  Capability  Level  5 

OEI  Capability  Level  3 

IT  Capability  Level  3 

ISM  Not  Rated 

QPM  Capability  Level  3 

VAL  Capability  Level  3 

CAR  Capability  Level  3 
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Reduce  complexity  and  ambiguity 
Provide  additional  guidance  where  needed 
Strengthen  appraisal  planning  and  conduct 
Strengthen  appraisal  reporting 
Define  appraisal  validity  period 
Strengthen  lead  appraiser  requirements 


CMMI 
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PDF  Complete. 


and  Expanded  Features 
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The  requirement  for  instruments  (e.g.,  questionnaires)  was  removed. 
Only  two  types  of  objective  evidence  are  now  required: 

A  documents 
A  interviews 

The  following  sections  in  MDD  were  revised: 

A  switched  2.2  Verify  and  Validate  Objective  Evidence  and  2.3  Document 
Objective  Evidence  so  that  the  order  of  tasks  reflects  the  natural  order  of 
conducting  an  appraisal 

A  separated  Verify  Objective  Evidence  and  Validate  Preliminary  Findings  to 
better  describe  each  process 
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The  use  of  the  term  instantiation  was  changed: 

A  Instantiation  is  now  defined  as  rthe  implementation  of  a  model  practice 
used  in  the  appropriate  context  within  the  boundaries  of  an  organizational 
unit. 6 

A  The  word  hnstantiationofor  project  and  organizational-wide  entities  was 
replaced  with  rprojectoor  Support  group. 6 
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The  rating  Not  Rated  was  clarified: 

A  Process  areas  outside  of  the  model  scope  are  rated  as  Out  of  Scope.  For 
example,  for  a  maturity  level  3  appraisal,  maturity  level  4  and  5  process 
areas  are  rated  as  Out  of  Scope. 

A  For  process  areas  that  have  insufficient  data  to  be  rated,  the  rating  is  Not 
Rated. 

A  Process  areas  in  the  model  scope,  but  outside  the  organizational  scope  are 
rated  as  Not  Applicable.  The  only  process  area  that  can  be  Not  Applicable 
is  SAM  (as  determined  by  the  appraisal  team). 

The  practice  characterization  tables  were  revised: 

A  clarified  the  use  of  virtual  versus  live  interviews 

A  changed  rface-to-face  interviewsoto  rbral  interviewso 
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Guidance  for  readiness  reviews  was  revised  to  include  team  and  logistical 
readiness. 

Additional  guidance  was  provided  for  using  virtual  methods  (e.g.,  for 
interviews  and  briefings). 

Additional  guidance  was  provided  for  using  alternative  practices  (i.e., 
Appendix  C:  Alternative  Practice  Identification  and  Characterization 
Guidance). 
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Organizational  unit  sampling  was  revised  to* 

A  strengthen  parameters  and  limits  for  organizational  sampling  (e.g., 
identifying  a  minimum  number  of  focus  projects) 

A  include  additional  criteria  for  reporting  sampling  decisions 

The  Conduct  Appraisal  Phase  must  now  be  complete  within  90  days. 

Appraisal  team  members  are  now  required  to  sign  final  findings. 

*Changes  to  address  sampling  were  extensive.  Refer  to  the  MDD  for  details. 
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The  Appraisal  Disclosure  Statement  (ADS)  now  requires  the  following 
information. 

Organizational  sampling  criteria  and  decisions  (e.g.,  projects  included, 
projects  excluded,  percentage  of  organization  represented) 

Basis  for  maturity/capability  level  4  and  5  appraisal  results 

A  subprocesses  statistically  managed 

A  mapping  of  these  subprocesses  to  quality  and  process-performance 
objectives 

Signatures  of  both  the  lead  appraiser  and  sponsor 

A  The  lead  appraiser  affirms  that  the  appraisal  scope  is  representative  of  the 
organizational  unit. 

A  The  sponsor  affirms  the  accuracy  of  the  ADS  and  authorizes  the  SEI  to 
conduct  any  audits  deemed  necessary. 
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VI  .2  appraisal  results  are  valid  for  a  maximum  of  3  years  from  the  date  of 
the  ADS. 

VI  .1  appraisals  will  expire  on  August  31 , 2007  or  3  years  after  the  date  the 
appraisal  was  conducted,  whichever  is  later. 
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Prior  to  conducting  a  vl  .2  SCAMPI  appraisal,  the  following  must  occur: 

A  Current  candidate  and  authorized  lead  appraisers  and  team  leaders  must 
complete  CMMI  vl  .2  Upgrade  Training. 

A  Candidate  and  authorized  lead  appraisers  must  attend  SCAMPI  Face-to- 
Face  Training. 

A  Those  who  want  to  conduct  vl  .2  SCAMPI  level  4  or  5  appraisals  must  be 
certified.  Certification  requirements  address  the  following: 

5  education,  training,  and  experience  in  level  4  and  5  concepts 

5  completion  of  an  oral  exam 
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The  SCAMPI  A  appraisal  method  was  revised  based  on  change  requests 
received  to 

A  reduce  complexity  and  ambiguity 

A  provide  additional  guidance  where  needed 

A  strengthen  appraisal  planning  and  conduct 

A  strengthen  appraisal  reporting 

A  define  the  appraisal  validity  period 

A  strengthen  lead  appraiser  requirements 

The  changes  are  intended  to  make  appraisals  more  accurate,  reliable,  and 
efficient. 
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Upgrade  Training 


SCAMPI  Lead 
AppraiserSM  Training 


SCAMPISM  B  and  C 
Team  Leader 
Training 


Instructor 

Training 
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The  following  courses  have  all  been  updated  to  address  change  requests 
and  CMMI  Product  Suite  vl.2  changes: 

A  Introduction  to  CMMI 

A  Intermediate  Concepts  of  CMMI 

A  CMMI  Instructor  Training 

A  SCAMPI  Lead  Appraiser  Training 

A  SCAMPI  B  and  C  Team  Leader  Training 

CMMI  vl  .2  Upgrade  Training  was  also  developed  to  help  users  move  from 
vl.1  to  vl.2,  an  online  course  with  potential  SEI  Partner  assistance 
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The  construction  and  format  of  examinations  have  changed,  vl  .1  tests 
were  largely  short  answer  tests  that  were  the  same  for  all  students. 

For  vl  .2,  tests  will  be  generated  from  an  item  bank  and  now  will  be 
multiple  choice.  CMMI  vl.2  Upgrade  Training  for  Instructors,  Lead 
Appraisers,  and  Team  Leaders  is  the  first  course  to  use  this  approach.  The 
Intermediate  Concepts  of  CMMI  and  Instructor  Training  will  follow. 
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This  new  approach,  using  an  item  bank  and  multiple  choice  questions, 
allows  multiple  versions  of  examinations  that  can  be  constructed  more 
easily: 

A  The  sequence  of  multiple  choice  responses  can  vary  from  test  to  test. 

A  The  order  of  questions  can  vary  from  test  to  test. 

A  The  selection  of  questions  can  vary  from  test  to  test,  but  cover  the  same 
categories. 

This  new  approach  allows  the  SEI  to 

A  add,  modify,  and  delete  questions  from  the  test  more  easily 
A  better  evaluate  the  student®  knowledge 
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Improved  architecture  will  allow  post-Vi. 2  expansion. 

A  Extensions  of  the  life  cycle  (Services,  Outsourcing/Acquisition)  could 
expand  use  of  a  common  organizational  framework: 

6  allows  coverage  of  more  of  the  enterprise  or  potential  partnering 
organizations 

6  adapts  model  features  to  fit  non-developmental  efforts  (e.g.,  CMMI 
Services,  CMMI  Acquisition) 
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CMMI-Dev 
provides  guidance 
for  measuring, 
monitoring  and 
managing 
development 
processes 


CMMI-SVC 


CMMI-DEV 
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CMMI-ACQ 
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acquisition 
leadership 
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Provide  a  /process  toolboxofor  the  acquirer 

A  Include  practical  guidance  on  how  to  recognize  the  real  practitionerse 

A  Encourage  the  use  of  capability  and  maturity  profiles  vice  "single  level" 
approach 

A  Improve  acquisition  organizations'  understanding  of  the  meaning  of  high 
maturity  (levels  4  and  5)  and  equivalent  staging 

A  Include  multiple  tools  and  guidance  that  may  be  used  throughout  the 
acquisition  lifecycle 
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Improve  acquisition  office  operating  practices 

A  Improve  Reviews'!  documents,  PMRs,  PDRs,  CDRse 

A  Improve  specific  areas:  risk  mgt,  requirements  mgt,  configuration  control, 
contracting  actions  (including  source  selection) 

A  Improve  communications 

A  Create  a  strategic  rhythmd 

A  Facilitate  synergy  between  program  segments/organizations,  and  even 
among  nsystems  of  systemso 

Facilitate  supplier  processes 

A  Oversight/Insight  into  supplier  processes 

A  Encourage  strategic  acquirer-supplier  teamwork  that  may  last  for  years 
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Random  motion  i  lots  of  energy,  not 
much  progress 

No  teamwork  T  individual  effort 
Frequent  conflict 

You  never  know  where  youdl  end  up 


Directed  motion  i  every  step  brings 
you  closer  to  the  goal 

Coordinated  efforts 

Cooperation 

Predictable  results 


Processes  can  make  the  difference  for  Developers  and 

Acquirers . 
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What  are  the  key  activities  you 
perform  when  you  acquire  systems? 


Requirements 

Management 

Configuration 

Management 


Risk  Program 

Management  Integration 


Verification 

and 

Validation 


Project 

Planning 


Need  to  counter  these  attitudes: 

I'd  rather  have  it  wrong  than  have  it  late.”  —  Industry  senior 

manager 

“Ad  hoc,  catch  as  you  can... that’s  our  motto.” 

“We  do  not  work  oroblems  until  thev’re  unrecoverable.” 
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Need 
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CMMI  for  Acquisition 

Acquisition 

Planning 

RFP 

Prep. 

Solicitation 

Source 

Selection 

Program  Leadership 
Insight  /  Oversight 

System 

Acceptance 

Transition 
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Plan 

Design 

Develop 

Integrate 
&  Test 

Deliver 

Developer 


New  Capability 
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The  Product  Suite  consists  of: 

ACMMI  for  Development 

AStandard  CMMI  Appraisal  Method  for 
Process  lmprovementsm  (SCAMPIsm) 

ATraining  and  Education 

ALicensing  Opportunities 

Aand  now  CMMI  for  Acquisition 
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Model  must  explicitly  apply  to  the  acquisition  of  a  wide  range  of  both 
products  and  services  (From  IT  outsourcing  to  DoD  acquisition  of  a 
weapon  system) 


Applicable  internationally  -  recognized  references  and  glossary  terms 
added,  e.g.,  service  level  measurement 


Model  must  apply  to  acquisition  organizations  from  commercial 
industry  to  government  agencies,  both  large  and  small 
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CMMI 


Organization 

Name 

Office  of  the  Secretary  of  Defense 

Kristen  Baldwin 

Navy 

Katie  Smith 

Air  Force 

Bob  Swarz 

Army 

Larry  Osiecki 

Defense  Contract  Management  Command 

Guy  Mercurio 

Missile  Defense  Agency 

Mike  Smith 

Government  Accounting  Office 

Madhav  Panwar 

General  Motors 

Rich  Frost 

National  Defense  Industrial  Association 

Bob  Rassa 
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CMMI-SG 

T 


Advisory  Board 


-  OUSD  (AT&L) 
-  Services  (3) 

-  MDA 

-  DCMA 

-  GAO 

-  GM 

-  NDIA  \/ 


Model  Team 
-  CMMI  Architect 
-  SEI 

-  Acquisition 
Experts 

(Services,  DAU, 
Commercial  and 
Defense 
Industry) 


Initial  Draft  CMMI-ACQ  C 


(requirements/design*) 


*  The  initial  draft  CMMI-ACQ 
model  requirements  and 
design  were  developed 
using  the  CMM  for 
Development  vl.2  model  as 
the  core 


(pilot/review/revise, 
making  changes  to  be 
consistent  with  1.2) 


|QA,  assurance] 
pilot 


Stakeholder  Community 
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:Q 


Key  Acquisition  Processes 


CMMI 


*based  on  initial  CMMI-ACQ  model  developed  by  General  Motors/SEI 


— CMMI  Version  1.2  and  Beyond 

^511=^  Software  Engineering  Institute  CarnegieMellon  Phillips,  November  12,  2007  57 

—  ©2007  Carnegie  Mellon  University 


CMMI 


|  APDF 

■»  Complete 


Your  complimentary 
use  period  has  ended. 
Thank  you  for  using 
PDF  Complete. 


and  Expanded  Features 


ining 


A  Phased  approach  will  be  used  for  ACQ  training 

A  Initial  training  for  CMMI-ACQ  will  be  face-to-face 

6  Assumes  the  completion  of  existing  Introduction  to  CMMI  training 
6  One-day  course  will  address  ACQ  concepts 

o  Pilot  offering  in  November 
o  First  public  offering  in  December 
o  Licensing  opportunity  will  be  available 

A  On-line  ACQ  upgrade  will  be  developed 

A  A  3-day  Introduction  to  CMMI  course  for  Acquisition  may  be  developed. 

A  In  FY  08,  the  Introduction  to  CMMI  course  will  be  revised  to  accommodate 
the  rmultiple  modeloapproach: 

6  16  CMMI  Model  Foundation  (CMF)  process  areas  will  be  taught  first 

s  Additions  for  the  nareas  of  interesto(development,  acquisition)  will  be  provided  to  meet  audience  needs 

A  Other  CMMI  courses  (e.g.,  Intermediate  and  instructor  training)  will  be 
updated  to  include  ACQ  material 
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‘Organizing  &  Managing 
the  Work” 


Managing 
the  Project 


Project  and 
Organizational 
Support 


◄ — ► 


'‘Providing  Infrastructure 
for  Projects 
&  Organizations” 


Doing  the  Work  of  the  Organization 


Product 
Acquisition  1 


Product 
Acquisition  2 


‘Understanding 
the  Work”  ' 


‘Performing 
the  Work” 


Managing 

Quantitatively 


1 Adding  Quantitative 
Management 
Capability  to  Other 
Management 
Approaches’ 


Product  Acquisition  1 

‘‘Enabling 

Product  Acquisition  2 

ARD,  SSAD 

Improvement 

AM,  ATM,  AVAL,  AVER 

of  the  Work” 
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A  Assure  model  understanding  before  allowing  benchmark 
claims 

6  Encourage  Class  B  and  C  appraisals  for  six  months 
6  Uncover  appraisal  issues  in  a  less  intense  environment 

A  Allow  time  to  align  appraisals  with  the  new  certification 
system  for  Lead  Appraisers  (March  08) 

A  Both  Lead  Appraisers  and  Instructors  must  pass  a 
qualification  test  for  CMMI-ACQ 

A  CMMI  development  and  governance  bodies  are 

considering  experience  requirements  for  Lead  Appraisers 
and  Instructors 
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SA-CMM 
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n  for  V2.0 


A  VI  .2  concentrated  on  the  project  or  program  level 
acquisition  best  practices 

A  V2.0  will  add  more  of  the  enterprise/organization  level  best 
practices  for  acquisition 

s  Address  enterprise  level  acquisition  strategies,  e.g., 
preferred  supplier  strategies 

6  Address  the  Program  Executive  Office  level 

6  Address  incorporation  of  lessons  learned  from  acquisition 
project  into  acquisition  management  practices 

A  V2.0  will  also  benefit  from  change  requests  issued  from 
lessons  learned  using  the  model  globally 
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resentation:  PAs  by  Categories 

Category 

Process  Areas 

Process 

Management 

Organizational  Process  Focus 

Organizational  Process  Definition  +IPPD 

Organizational  Training 

Organizational  Process  Performance 

Organizational  Innovation  and  Deployment 

Project 

Management 

Project  Planning 

Project  Monitoring  and  Control 

Supplier  Agreement  Management 

Integrated  Project  Management  +IPPD 

Risk  Management 

Quantitative  Project  Management 

Engineering 

Requirements  Management 

Requirements  Development 

Technical  Solution 

Product  Integration 

Verification 

Validation 

Support 

Configuration  Management 

Process  and  Product  Quality  Assurance 

Measurement  and  Analysis 

Decision  Analysis  and  Resolution 

Causal  Analysis  and  Resolution 
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ntation:  PAs  by  Maturity  Level  cmmi. 


5  Optimizing 

Continuous 

Process 

Improvement 

Organizational  Innovation  and  Deployment 

Causal  Analysis  and  Resolution 

4  Quantitatively 
Managed 

Quantitative 

Management 

Organizational  Process  Performance 

Quantitative  Project  Management 

3  Defined 

Process 

Standardization 

Requirements  Development 

Technical  Solution 

Product  Integration 

Verification 

Validation 

Organizational  Process  Focus 

Organizational  Process  Definition  +IPPD 
Organizational  Training 

Integrated  Project  Management  +IPPD 

Risk  Management 

Decision  Analysis  and  Resolution 

2  Managed 

Basic 

Project 

Management 

Requirements  Management 

Project  Planning 

Project  Monitoring  and  Control 

Supplier  Agreement  Management 

Measurement  and  Analysis 

Process  and  Product  Quality  Assurance 

Configuration  Management 

1  Initial 
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presentation:  PAs  by  Maturity  Level 


Maturity  Level 

Process  Areas 

Optimizing 

Causal  Analysis  and  Resolution 

Organizational  Innovation  and  Deployment 

Quantitatively 

Managed 

Quantitative  Project  Management 

Organizational  Process  Performance 

Defined 

Organizational  Process  Focus 

Organizational  Process  Definition 

Organizational  Training 

Integrated  Project  Management 

Risk  Management 

Acquisition  Technical  Management 

Acquisition  Verification 

Acquisition  Validation 

Decision  Analysis  and  Resolution 

Managed 

Acquisition  Requirements  Development 

Agreement  Management 

Project  Planning 

Project  Monitoring  and  Control 

Requirements  Management 

Configuration  Management 

Process  and  Product  Quality  Assurance 

Measurement  and  Analysis 

Solicitation  and  Supplier  Agreement  Development 
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CMMI 


performed  vs.  managed 

the  extent  to  which  the  process  is  planned;  performance  is 
managed  against  the  plan;  corrective  actions  are  taken  when 
needed 

managed  vs.  defined 

the  scope  of  application  of  the  process  descriptions,  standards,  and 
procedures  (i.e.,  project  vs.  organization) 

defined  vs.  quantitatively  managed 

the  predictability  of  process  performance 


quantitatively  managed  vs.  optimizing 

whether  the  process  is  continually  improved  by  addressing 
common  causes  of  process  variation 
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CMMI 


Levels  are  used  in  CMMI  to  describe  an  evolutionary  path  for  an 
organization  that  wants  to  improve  the  processes  it  uses  to  develop  and 
maintain  its  products  and  services. 

CMMI  supports  two  improvement  paths: 

A  continuous  -  enabling  an  organization  to  incrementally  improve  processes 
corresponding  to  an  individual  process  area  (or  set  of  process  areas) 
selected  by  the  organization 

A  staged  -  enabling  the  organization  to  improve  a  set  of  related  processes  by 
incrementally  addressing  successive  predefined  sets  of  process  areas 
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CMMI 


GP2.1  through  GP3.2 
All  ML2,  ML3,  ML4, 
and  ML5  PAs 

GP2.1  through  GP3.2 
All  ML2,  ML3,  and 
ML4  PAs 


GP2.1  through  GP3.2 
All  ML2  and  ML3  PAs 


GP2.1  through  GP2.10 
All  ML2  PAs 


Prevent  defects;  proactively  improve;  insert  and  deploy 
innovative  technology 


Measure  process  performance;  stabilize  process  and 
control  charts;  deal  with  causes  of  special  variations 


Tailor  the  project’s  process  from  organization’s 
standard  processes;  understand  processes  qualitatively; 
ensure  that  projects  contribute  to  organization  assets 


Adhere  to  policy;  follow  documented  plans  and  processes; 
apply  adequate  resources;  assign  responsibility  and 
authority;  train  people;  apply  CM;  monitor,  control,  and 
evaluate  process;  identify  and  involve  stakeholders; 
review  with  management 

Processes  are  ad  hoc  and  chaotic 
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KEY: 
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Generic  Goals 


meric  Goals  and  Practices 


CMMI 


Generic  Practices 


GG  Achieve 

Specific  Goals 

GP  .  Perform  Specific  Practices 

GG  Institutional!  ea 
Managed  Process 

GP  .  Esta  lish  an  Organi  ational  Policy 

GP  .  Plan  the  Process  i 

GP  .  Provide  Resources 

GP  .  Assign  Responsi  ility 

GP  .  Train  People 

GP  .  Manage  Configurations 

GP  .  Identify  and  Involve  Relevant  Stakeholders 

GP  .  Monitor  and  Control  the  Process 

GP  .  0  ectively  Evaluate  Adherence 

GP  .  Review  Status  with  igher  evel  Management 

GG  Institutional!  e  a 

Defined  Process 

GP  .  Esta  lish  a  Defined  Process 

GP  .  Collect  Improvement  Information 

GG  Institutional!  ea 
Quantitatively 

Managed  Process 

GP  .  Esta  lish  Quantitative  0  ectives  for  the  Process 

GP  .  Sta  ili  e  Su  process  Performance 

GG  Institutional!  ean 

Optimi  ing  Process 

GP  .  Ensure  Continuous  Process  Improvement 

GP  .  Correct  Root  Causes  of  Pro  lems 

Adapted  from 
Cepeda  Systems  & 
Software  Analysis,  Inc. 
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tion  Inputs 


CMMI 


Process 

Architecture 


Process  Needs 


Process  Descriptions, 
Procedures, 

|  Instructions  j 


Asset  Library 


Measurement 

Repository 


Process  Scope 
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Area  Category  Topics 


CMMI 


Process  Management 
Support 

Project  Management 

Engineering 

Acquisition 
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Process  Definition  +IPPD  -1 


Specific  Goal  Specific  Practice 


Establish  Organizational 
Process  Assets 


1.1  -  Establish  Standard  Processes 

1.2  -  Establish  Lifecycle  Model  Descriptions 

1.3  -  Establish  Tailoring  Criteria  and  Guidelines 

1.4  -  Establish  the  Organization’s  Measurement 

Repository 

1.5  -  Establish  the  Organization’s  Process  Asset 

Library 

1.6  -  Establish  Work  Environment  Standards 


Mdded  rand  work  environment  standardsoto  the  purpose  statement. 

Mdded  SP  1 .6  ("Establish  Work  Environment  Standards. o(This  practice 
is  new  to  CMMI.) 
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Process  Definition  +IPPD  -2 


Specific  Goal _ Specific  Practice _ 

Enable  IPPD  Management  2.1  -  Establish  Empowerment  Mechanisms 

2.2  -  Establish  Rules  and  Guidelines  for 

Integrated  Teams 

2.3  -  Balance  Team  and  Home  Organization 

Responsibilities 


Mdded  an  IPPD  Addition  to  OPD  (SG2  ("Enable  IPPD  Managementoand 
its  practices). 

ATo  emphasize  the  IPPD  Addition,  the  name  the  process  area  is  now 
("Organizational  Process  Definition  +IPPDoor  (OPD  +IPPD.0 
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Specific  Goal _ Specific  Practice _ 

Determine  Process  1.1  -  Establish  Organizational  Process 

Improvement  Opportunities  Needs 

1.2  -  Appraise  the  Organization’s  Processes 

1.3  -  Identify  the  Organization’s  Process 

Improvements 


AModified  the  purpose  statement  to  emphasize  deployment. 

ASP  1.2  fAppraise  the  organization®  processes  periodically  and  as 
needed  to  maintain  an  understanding  of  their  strengths  and 
weaknesses. ouses  organization®  processesoinstead  of  (processes  of 
the  organization. 6 
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CMMI 


Specific  Goal_ Specific  Practice 

Plan  and  Implement  2.1  -  Establish  Process  Action  Plans 

Process  2.2  -  Implement  Process  Action  Plans 

Improvements 


AModified  SG2  from  rPlan  and  Implement  Process  Improvement 
Activitiesoto  rPlan  and  Implement  Process  Improvements. 6 

AMoved  to  a  new  SG3  and  modified  what  were  SP  2.3  and  SP  2.4  in 
vl.1. 
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Specific  Goal  Specific  Practice 


Deploy  Organizational 

3.1 

Process  Assets  and 

Incorporate  Lessons 

3.2 

Learned 

3.3 

3.4 

-  Deploy  Organizational  Process 
Assets 

-  Deploy  Standard  Processes 

-  Monitor  Implementation 

-  Incorporate  Process-Related 
Experiences  into  the 
Organizational  Process  Assets 


Mdded  new  SG3,  (Deploy  Organizational  Process  Assets  and 
Incorporate  Lessons  Learned. 6 

AMoved  what  were  SP  2.3  and  SP  2.4  in  vl  .1  to  the  new  SG3  as  SP  3.1 
and  SP  3.4. 

Mdded  two  new  SPs:  SP  3.2  rDeploy  Standard  Processes, oand  SP  3.3 
rMonitor  Implementation. 6 
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CMMI 


SG  1:  Establish  an  Organizational  Training  Capability 
A  training  capability  that  supports  the  organization®  management  and 
technical  roles  is  established  and  maintained. 

SG  2:  Provide  Necessary  Training 

Training  necessary  for  individuals  to  perform  their  roles  effectively  is 
provided. 

The  process  area  also  has  generic  goals  to  support  institutionalization. 
Note  relationship  with 

A  Organizational  Training  GP  2.5 
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and  Deployment  Goals 


SG  1:  Select  Improvements 

Process  and  technology  improvements  that  contribute  to  meeting  quality 
and  process-performance  objectives  are  selected. 

SG  2:  Deploy  Improvements 

Measurable  improvements  to  the  organization®  processes  and 
technologies  are  continually  and  systematically  deployed. 

The  process  area  also  has  generic  goals  to  support  institutionalization. 

Note  relationship  with 

A  Organizational  Innovation  and  Deployment  GP  5.1 
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SG  1 :  Establish  Performance  Baselines  and  Models 

Baselines  and  models  that  characterize  the  expected  process  performance 

of  the  organization®  set  of  standard  processes  are  established  and 

maintained. 

The  process  area  also  has  generic  goals  to  support  institutionalization. 
Note  relationship  with 

A  Organizational  Process  Performance  j)  GP  4.1 
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CMMI 


SG  1 :  Establish  Baselines 

Baselines  of  identified  work  products  are  established. 

SG  2:  Track  and  Control  Changes 

Changes  to  the  work  products  under  configuration  management  are 
tracked  and  controlled. 

SG  3:  Establish  Integrity 

Integrity  of  baselines  is  established  and  maintained. 

The  process  area  also  has  generic  goals  to  support  institutionalization. 
Note  relationship  with 

A  Configuration  Management  S  S  GP  2.6 
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CMMI 


SG  1 :  Align  Measurement  and  Analysis  Activities 
Measurement  objectives  and  activities  are  aligned  with  identified 
information  needs  and  objectives. 

SG  2:  Provide  Measurement  Results 

Measurement  results  that  address  identified  information  needs  and 
objectives  are  provided. 

The  process  area  also  has  generic  goals  to  support  institutionalization. 
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and  Expanded  Features 


SG  1 :  Objectively  Evaluate  Processes  and  Work  Products 
Adherence  of  the  performed  process  and  associated  work  products  and 
services  to  applicable  process  descriptions,  standards,  and  procedures  is 
objectively  evaluated. 

SG  2:  Provide  Objective  Insight 

Noncompliance  issues  are  objectively  tracked  and  communicated,  and 
resolution  is  ensured. 

The  process  area  also  has  generic  goals  to  support  institutionalization. 
Note  relationship  with 
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CMMI 


SG  1 :  Evaluate  Alternatives 

Decisions  are  based  on  an  evaluation  of  alternatives  using  established 
criteria. 

The  process  area  also  has  generic  goals  to  support 
institutionalization. 
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CMMI 


SG  1 :  Determine  Causes  of  Defects 

Root  causes  of  defects  and  other  problems  are  systematically  determined. 
SG  2:  Address  Causes  of  Defects 

Root  causes  of  defects  and  other  problems  are  systematically  addressed 
to  prevent  their  future  occurrence. 

The  process  area  also  has  generic  goals  to  support  institutionalization. 
Note  relationship  with 

A  Causal  Analysis  and  Resolution  j)  GP  5.2 
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SG  1 :  Establish  Estimates 

Estimates  of  project  planning  parameters  are  established  and  maintained. 
SG  2:  Develop  a  Project  Plan 

A  project  plan  is  established  and  maintained  as  the  basis  for  managing  the 
project. 

SG  3:  Obtain  Commitment  to  the  Plan 

Commitments  to  the  project  plan  are  established  and  maintained. 

The  process  area  also  has  generic  goals  to  support  institutionalization. 
Note  relationship  with 

A  Project  Planning  ^  S  GP  2.2,  GP  2.7 
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CMMI 


SG  1 :  Monitor  Project  Against  Plan 

Actual  performance  and  progress  of  the  project  are  monitored  against  the 
project  plan. 

SG  2:  Manage  Corrective  Action  to  Closure 

Corrective  actions  are  managed  to  closure  when  the  project®  performance 
or  results  deviate  significantly  from  the  plan. 

The  process  area  also  has  generic  goals  to  support  institutionalization. 
Note  relationship  with 

A  Project  Monitoring  and  Control  P>  GP  2.8 
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Specific  Goal 

Specific  Practice 

Establish  Supplier 
Agreements 

1.1  -  Determine  Acquisition  Type 

1.2  -  Select  Suppliers 

1.3  -  Establish  Supplier  Agreements 

Satisfy  Supplier 
Agreements 

2.1  -  Execute  the  Supplier  Agreement 

2.2  -  Monitor  Selected  Supplier 

Processes 

2.3  -  Evaluate  Selected  Supplier  Work 
Products 

2.4  -  Accept  the  Acquired  Product 

2.5  -  Transition  Products 

AVI.1  SAM  SP2.1  rReview  COTS  Products, owas  eliminated.  (Identify 
candidate  COTS  products  that  satisfy  requirementsois  a  new 
subpractice  under  the  Technical  Solutions  Process  Area  SP1 .1 , 
(Develop  Alternative  Solutions  and  Selection  Criteria. 6 

ASP2.2  and  SP2.3  were  added  because  ISM  was  eliminated. 

AThe  purpose  of  SAM  was  also  updated. 
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Specific  Goal  Specific  Practice 

Use  the  Project’s  Defined  1.1  -  Establish  the  Project’s  Defined  Process 
Process 

1.2  -  Use  Organizational  Process  Assets  for 

Planning  Project  Activities 

1.3  -  Establish  the  Project’s  Work  Environment 

1.4  -  Integrate  Plans 

1.5  -  Manage  the  Project  Using  the  Integrated 

Plans 

1.6  -  Contribute  to  the  Organizational 

Process  Assets 


AModified  SP  1.1  from  ("Establish  and  maintain  the  project®  defined 
processoto  (Establish  and  maintain  the  project®  defined  process  from 
project  startup  through  the  life  of  the  project.o 

Mdded  SP  1.3  (Establish  the  Project®  Work  Environment. o(This  practice 
is  new  to  CMMI.) 
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Specific  Goal_ Specific  Practice 

Coordinate  and  Collaborate  2.1  -  Manage  Stakeholder  Involvement 
with  Relevant  Stakeholders 

2.2  -  Manage  Dependencies 

2.3  -  Resolve  Coordination  Issues 

Apply  IPPD  Principles  3.1  -  Establish  the  Project’s  Shared 

Vision 

3.2  -  Establish  the  Integrated  Team 

Structure 

3.3  -  Allocate  Requirements  to 

Integrated  Teams 

3.4  -  Establish  Integrated  Teams 

3.5  -  Ensure  Collaboration  among 

Interfacing  Teams 

AReduced  the  IPPD  Addition  to  one  goal  (SG3  fApply  IPPD  Principles^ 
and  its  practices. 

ATo  emphasize  the  IPPD  Addition,  the  name  of  this  process  area  is  now 
(Integrated  Project  Management  +IPPDoor  rlPM  +IPPD.0 
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CMMI 


A  IPM  SP  1.6  Establish  and  maintain  integrated  teams. 

A  The  project  is  managed  using  integrated  teams  that  reflect  the 
organizational  rules  and  guidelines  for  team  structuring  and  forming. 
The  project®  shared  vision  is  established  prior  to  establishing  the  team 
structure,  which  may  be  based  on  the  WBS.  For  small  acquirer 
organizations,  the  whole  organization  and  the  relevant  external 
stakeholders  can  be  treated  as  an  integrated  team. 

A  Integrated  team  members  must  understand  the  standards  for  work  and 
participate  according  to  those  standards. 

A  Structuring  the  integrated  teams  involves  defining  the  number  of  teams, 
the  type  of  each  team,  and  how  each  team  relates  with  the  others  in  the 
structure.  Forming  the  integrated  teams  involves  chartering  each  team, 
assigning  team  members  and  team  leaders,  and  providing  resources  to 
each  team  to  accomplish  its  work. 
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CMMI 


SG  1 :  Prepare  for  Risk  Management 
Preparation  for  risk  management  is  conducted. 

SG  2:  Identify  and  Analyze  Risks 

Risks  are  identified  and  analyzed  to  determine  their  relative  importance. 
SG  3:  Mitigate  Risks 

Risks  are  handled  and  mitigated,  where  appropriate,  to  reduce  adverse 
impacts  on  achieving  objectives. 

The  process  area  also  has  generic  goals  to  support  institutionalization. 
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SG  1 :  Quantitatively  Manage  the  Project 

The  project  is  quantitatively  managed  using  quality  and  process- 

performance  objectives. 

SG  2:  Statistically  Manage  Subprocess  Performance 

The  performance  of  selected  subprocesses  within  the  project®  defined 

process  is  statistically  managed. 

The  process  area  also  has  generic  goals  to  support  institutionalization. 
Note  relationship  with 

A  Quantitative  Project  Management  P>  GP4.1,GP4.2 
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Specific  Goal _ Specific  Practice _ 

Manage  Requirements  1.1  -  Obtain  an  Understanding  of 

Requirements 

1.2  -  Obtain  Commitment  to  Requirements 

1.3  -  Manage  Requirements  Changes 

1.4  -  Maintain  Bidirectional  Traceability  of 

Requirements 

1.5  -  Identify  Inconsistencies  Between 

Project  Work  and  Requirements 


AVI  .2  REQM  SP  1 .4  practice  statement  now  reads,  ("Maintain 
bidirectional  traceability  among  the  requirements  and  work  products. 6 

AProject  plans  are  no  longer  mentioned  in  this  SP  statement. 

AThe  description  of  bidirectional  traceability  is  improved  as  is  its 
definition  in  the  glossary. 
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Specific  Goal 

Specific  Practice 

Develop  Customer 
Requirements 

1.1  -  Elicit  Needs 

1.2  -  Develop  the  Customer  Requirements 

Develop  Product 
Requirements 

2.1  -  Establish  Product  and  Product 

Component  Requirements 

2.2  -  Allocate  Product  Component 

Requirements 

2.3  -  Identify  Interface  Requirements 

AFormer  base  practice  rCollect  Stakeholder  Needsois  eliminated  and 
former  advanced  practice,  rElicit  Needsois  kept. 

Alnformative  text  is  added  to  the  introductory  notes  about  applying  RD 
to  maintenance  projects. 
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Specific  Goal _ Specific  Practice _ 

Analyze  and  Validate  3.1  -  Establish  Operational  Concepts  and 

Requirements  Scenarios 

3.2  -  Establish  a  Definition  of  Required 

Functionality 

3.3  -  Analyze  Requirements 

3.4  -  Analyze  Requirements  to  Achieve 

Balance 

3.5  -  Validate  Requirements 


AMaterial  from  VI. 1  TS  SP  1.2,  rEvolve  Operational  Concepts  and 
Scenarios, ois  incorporated  into  RD  SP  3.1. 

AMaterial  from  VI. 1  RD  SP  3.5-1,  (Validate  Requirements, oand  RD 
SP  3.5-2,  (Validate  Requirements  with  Comprehensive  Methodso 
were  consolidated  into  a  single  practice. 
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Specific  Goal 


Specific  Practice 


Select  Product-Component  1.1  -  Develop  Alternative  Solutions  and 


Solutions 


Selection  Criteria 

1.2  -  Select  Product-Component  Solutions 


Avi.1  TS  SP  1.1-1,  (Develop  Alternative  Solutions  and  Selection 
Criteria, oand  TS  SP  1.1-2,  rDevelop  Detailed  Alternative  Solutions  and 
Selection  Criteriaoare  consolidated  into  a  single  practice. 

Arldentify  candidate  COTS  products  that  satisfy  requirementsois  a  new 
subpractice  under  SP1 .1 . 

AVI.1  TS  SP  1.2  (Evolve  Operational  Concepts  and  Scenariosois 
incorporated  into  RD  SP  3.1,  ("Establish  Operational  Concepts  and 
Scenarios. 6 
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Specific  Goal 

Specific  Practice 

Develop  the  Design 

2.1  -  Design  the  Product  or  Product 

Component 

2.2  -  Establish  a  Technical  Data  Package 

2.3  -  Design  Interfaces  Using  Criteria 

2.4  -  Perform  Make,  Buy,  or  Reuse  Analyses 

Implement  the 

3.1  -  Implement  the  Design 

Product  Design 

3.2  -  Develop  Product  Support 
Documentation 

Avi.1  TS  SP  2.3-1,  ("Establish  Interface  Descriptions, band  TS  SP  2.3- 
3,  ("Design  Interfaces  Using  Criteriaoare  consolidated  into  a  single 
practice. 
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SG  1 :  Prepare  for  Product  Integration 
Preparation  for  product  integration  is  conducted. 

SG  2:  Ensure  Interface  Compatibility 

The  product  component  interfaces,  both  internal  and  external,  are 
compatible. 

SG  3:  Assemble  Product  Components  and  Deliver  the  Product 
Verified  product  components  are  assembled  and  the  integrated,  verified, 
and  validated  product  is  delivered. 

The  process  area  also  has  generic  goals  to  support  institutionalization. 
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Specific  Goal 

Specific  Practice 

Prepare  for  Verification 

1.1  -  Select  Work  Products  for  Verification 

1.2  -  Establish  the  Verification 

Environment 

1.3  -  Establish  Verification  Procedures 

and  Criteria 

Perform  Peer  Reviews 

2.1  -  Prepare  for  Peer  Reviews 

2.2  -  Conduct  Peer  Reviews 

2.3  -  Analyze  Peer  Review  Data 

ANo  changes  to  SGI ,  SG2,  or  their  practices. 
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Specific  Goal 

Verify  Selected  Work 
Products 


Specific  Practice 

3.1  -  Perform  Verification 

3.2  -  Analyze  Verification  Results 


AThe  phrase  rand  identify  corrective  actionowas  deleted  from  both 
the  title  and  statement  of  SP  3.2  Analyze  Verification  Results. 
(Corrective  action  is  handled  in  PMC  SG2,  rManage  Corrective 
Action  to  Closure.) 
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Specific  Goal _ Specific  Practice _ 

Prepare  for  Validation  1.1-  Select  Products  for  Validation 

1.2  -  Establish  the  Validation  Environment 

1.3  -  Establish  Validation  Procedures  and 

Criteria 


Validate  Product  or  2.1  -  Perform  Validation 

Product  Components  2.2  -  Analyze  Validation  Results 


ANotes  were  added  to  VAL  to  stress  that  validation  activities  are 
performed  incrementally  and  involve  relevant  stakeholders. 

AThe  phrase  rand  identify  issuesowas  deleted  from  the  statement  of  SP 

2.2  rAnalyze  Validation  Resultsoto  maintain  parallelism  with  VER  SP 

3.2  fAnalyze  Verification  Results. 6 
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SG  1 :  Develop  Customer  Requirements 

Stakeholder  needs,  expectations,  constraints,  and  interfaces  are  collected 
and  translated  into  customer  requirements. 

SG  2:  Develop  Contractual  Requirements 

Customer  requirements  are  refined  and  elaborated  to  develop  contractual 
requirements. 

SG  3:  Analyze  and  Validate  Requirements 
The  requirements  are  analyzed  and  validated. 
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SG  1 :  Prepare  for  Solicitation  and  Supplier  Agreement 
Development 

Preparation  for  solicitation  and  supplier  agreement  is  performed. 

SG  2:  Select  Suppliers 

Suppliers  are  selected  based  on  an  evaluation  of  their  ability  to 
meet  the  specified  requirements  and  established  criteria. 

SG  3:  Establish  Supplier  Agreements 

Supplier  agreements  are  established  and  maintained. 
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CMMI 


SG  1:  Evaluate  Technical  Solutions 

Supplier  technical  solutions  are  evaluated  to  confirm 
that  contractual  requirements  continue  to  be  met. 

SG  2:  Perform  Interface  Management 
Selected  interfaces  are  managed. 
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Manage 
Supplier 
Agreements 


Revised  Supplier 
Agreement 


Approval  Reports 
Discrepancy  Reports 
Acceptance  Reports 


Escalated  Issues 
Resolutions 


Approved  Payment 
Invoices 
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SG  1 :  Prepare  for  Verification 

Preparation  for  verification  is  conducted. 

SG  2:  Perform  Peer  Reviews 

Peer  reviews  are  performed  on  selected  work  products. 

SG  3:  Verify  Selected  Work  Products 

Selected  work  products  are  verified  against  their  specified 
requirements. 
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SG  1 :  Prepare  for  Validation 

Preparation  for  validation  is  conducted. 

SG  2:  Validate  Selected  Products  and  Product 
Components 

Selected  products  and  product  components  are 
validated  to  ensure  that  they  are  suitable  for  use 
in  their  intended  operating  environment. 
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The  key  additions  to  the  CMF  include  the  following: 

A  Acquisition  Strategy 

A  Transition  to  Operations  and  Support 

A  Integrated  Product  and  Process  Development  (Teaming) 

There  are  informative  materials  unique  to  the  Acquisition 
Constellation  in  every  process  area. 
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A  Acquisition  strategy  -  Planning  begins  with  the  acquisition  strategy  that  provides  the 
framework  for  the  acquisition  project  and  its  plans. 

A  PP  SP  1.1  Establish  and  maintain  the  acquisition  strategy. 

A  The  strategy  has  the  following  attributes: 

A  used  to  focus  on  specifying  customer  and  contractual  requirements  that  express 
customer  value  in  the  Acquisition  Requirements  Development  process  area 
practices. 

A  is  the  business  and  technical  management  framework  for  planning,  executing,  and 
managing  agreements  for  a  project. 

A  relates  to  the  objectives  for  the  acquisition,  the  constraints,  availability  of  resources 
and  technologies,  consideration  of  acquisition  methods,  potential  supplier  agreement 
types,  terms  and  conditions,  accommodation  of  business  considerations, 
considerations  of  risk,  and  support  for  the  acquired  product  over  its  lifecycle. 

A  reflects  the  entire  scope  of  the  project. 

A  encompasses  the  work  to  be  performed  by  the  acquirer  and  the  work  to  be 

performed  by  the  supplier,  or  in  some  cases  multiple  suppliers,  for  the  full  lifecycle  of 
_ the  product _ 
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Transition  to  operations  and  support  includes  the  approach 
for  introducing  and  maintaining  the  readiness,  sustainment, 
and  operational  capability  of  the  product(s)  delivered  by  the 
supplier. 

A  PP  SP  2.7  -  Plan  for  transition  to  lifecycle  operations  and 
support  for  the  product. 

A  PMC  SP  1 .8  -  Monitor  the  transition  to  operations  and 
support. 

Typically,  the  supplier  has  a  role  in  integrating  and  packaging  the  products 
and  prepares  for  the  transition  to  operations  and  support,  including  support 
for  business  user  acceptance.  The  acquirer  monitors  supplier  activities. 
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Integrated  Teams-  The  project  is  managed  using  integrated  teams  (IPM  SP 
1 .6)  that  reflect  the  organizational  rules  and  guidelines  (OPD  SP  1 .7)  for  team 
structuring  and  forming. 

AOPD  SP  1.7  Establish  and  maintain  organizational  rules  and  guidelines  for  the 
structure  and  operation  of  integrated  teams. 

A  In  an  acquisition  organization,  integrated  teams  are  useful  not  just  in  the 
acquirer®  organization,  but  between  the  acquirer  and  supplier,  and 
among  the  acquirer,  supplier  and  other  relevant  stakeholders  as 
appropriate.  Integrated  teaming  may  be  especially  important  in  a 
system  of  systems  environment. 

A  Operating  rules  and  guidelines  for  integrated  teams  define  and  control 
how  teams  are  created  and  how  they  interact  to  accomplish  objectives. 
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For  more  information  about  CMMI 

A  http://www.sei.cmu.edu/cmmi/  (main  CMMI  site) 

Other  Web  sites  of  interest  include 

A  http://seir.sei.cmu.edu/seir/  (Software  Engineering  Information  Repository) 

A  http://dtic.mil/ndia  (annual  CMMI  Technology  Conferences) 

A  http://seir.sei.cmu.edu/pars  (publicly  released  SCAMPI  appraisal  summaries) 
A  https  ://bscw.  sei .  cm  u  .ed  u/pu  b/bscw.  eg  i/0/79783 
Or,  contact 

SEI  Customer  Relations 

Phone:  412/ 268-5800 

Email:  customer-relations@sei.cmu.edu 
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Process-performance  baselines  are  built  from  project  data. 


Projects  use  the  organization's  process-performance  baselines  in 
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PPBs  are  derived  by  analyzing  the 
collected  measures  to  establish  a 
distribution  and  range  of  results  that 
characterize  the  expected  performance  for 
selected  processes  when  used  on  any 
individual  project  in  the  organization. 


Boxplot  of  web  %  vs  Call  Type 


Change  Complaint  Problem  Question 

Call  Type 


UCL  =  31 .6 


CL  =  20.04 


LCL  =  8.49 


Summary  for  Defect  Density 


95%  Confidence  Intervals 


Mean 

Median 


Anderson-Darling  Normality  Test 

A-Squared 

35.17 

P-Value  < 

0.005 

Mean 

0.98220 

StDev 

0.90085 

Variance 

0.81154 

Skewness 

1.29664 

Kurtosis 

1.47988 

N 

1000 

Minimum 

0.00057 

1st  Quartile 

0.27760 

Median 

0.73288 

3rd  Quartile 

1.41156 

Maximum 

4.53196 

95%  Confidence  Interval  for  Mean 

0.92630 

1.03810 

95%  Confidence  Interval  for  Median 

0.66187 

0.79057 

95%  Confidence  Interval  for  StDev 

0.86303 

0.94217 
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A  To  determine  whether  a  process  change  (improvement  or 
degradation)  has  occurred  by  comparing  the  before  and  after 
baselines. 

A  To  determine  whether  or  not  a  new  sample  of  data  representing 
the  current  process  behavior  is  with  previous  process 

behavior  (e.g.,  a  historical  baseline). 

A  To  determine  whether  or  not  stratification  or  segmentatior  of  the 

data  is  appropriate  by  comparing  a  baseline  from  each  segment 
or  strata  layer  to  other  segments  or  strata  layers. 

A  To  enable  comparisons  between  projects  or 

organizations. 
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The  organization  uses  PPMs 

A  for  estimating,  analyzing,  and  predicting  the  process  performance  of 
processes  in  the  organization's  set  of  standard  processes. 

A  to  assess  the  (potential)  return  on  investment  of  process 
improvement  activities. 


Projects  use  PPMs 

A  for  estimating,  analyzing,  and  predicting  the  performance  of  their 
defined  processes. 

A  for  selecting  which  processes  to  include  in  the  project®  defined 
process. 
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A  The  essential  ingredients  of  process-performance  models  include  the 
following: 

A  The  models  relate  the  behavior  or  circumstance  of  a  process  or 
subprocess  to  an  outcome. 

A  The  models  predict  future  outcomes  based  on  possible  or  actual 
changes  to  factors  (e.g.,  support  rwhat-ifoanalysis). 

A  The  models  use  factors  from  one  or  more  subprocesses  to  conduct 
the  prediction.  These  factors  are  preferably  controllable  so  that 
projects  may  take  action  to  influence  outcomes. 

A  The  models  are  statistical  or  probabilistic  in  nature  rather  than 
deterministic  (e.g.,  the  models  account  for  statistical  variation  like 
QPM  does;  the  models  depict  uncertainty  in  the  factors  and  predict 
the  uncertainty  or  range  of  values  in  the  outcome). 
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A  High  maturity  organizations  generally  possess  a  collection  of  process- 
performance  models  that  go  beyond  earned  value  measures  that  predict 
cost  and  schedule  variance. 

A  Specifically,  the  models  predict  quality  and  performance  outcomes  from 
factors  related  to  one  or  more  subprocesses  involved  in  the 
development,  maintenance,  service,  or  acquisition  processes  performed 
by  the  projects.  Example  outcomes  include  the  following: 

A  schedule,  effort,  or  cost  variance 

A  reliability  of  delivery  to  the  customer 

A  defect  identification  and  removal  rates 

A  customer  satisfaction 

A  success  indicators  identified  by  the  organization  or  project 
A  a  combination  of  these  outcomes 
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A  Lifecycle  phase  subprocesses 

Consider  lifecycle  phases  such  as  the  following:  requirements,  architecture, 
design,  code,  and  test.  Example  subprocesses  include  requirements  elicitation, 
requirements  allocation,  architecture  selection,  and  design,  code  and  test 
review.  Example  attributes  of  these  subprocesses  include  cycle  time,  quality 
performance  or  defect  density,  productivity,  staff  attributes,  and  risk  indices. 

A  Keep  in  mind  attributes  such  as  downtime  of  parts  of  the  project  environment 
(e.g.,  computing  resources,  test  equipment,  and  specialized  tools  and 
compilers). 

A  Inspection  and  peer  review  subprocesses 

Consider  subprocesses  that  are  important  to  understand  quality  and  therefore 
are  important  to  your  business  such  as  preparation,  meeting  conduct  and 
review.  Example  attributes  of  these  subprocesses  include  preparation  times, 
review  rates,  and  defect  densities. 

A  Other  subprocesses 

Consider  subprocesses  (e.g,  supplier  agreement  development,  supplier 
monitoring,  customer  interaction,  partner  development)  that  involve  responding 
to  inquiries  or  actions  related  to  key  interfaces  with  suppliers,  customers,  and 
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A  Olympic  swimmers  use  process-performance  models  to  evaluate  their 
overall  race  time.  With  years  of  experience,  they  have  identified  several 
key  subprocesses  that  dominate  the  overall  race  time: 

A  the  time  off  the  blocks  at  the  start  of  the  race 
A  the  time  it  takes  to  complete  a  turn  at  the  end  of  the  pool 

A  By  controlling  and  managing  these  times,  Olympic  swimmers  have 
attained  world  class  performance. 
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Many  changes  were  made  to  the  CMMI  models  to  improve  quality.  The 
major  changes  include 

A  name  changed  to  rCMMI  for  Developmento 

A  both  representations  in  one  document 

A  amplifications  improved;  added  hardware  amplifications 

A  common  features  and  advanced  practices  eliminated 

A  SS  addition  eliminated;  ISM  brought  into  SAM 

A  guidelines  for  mot  applicableoprocess  areas  clarified 

A  overview  and  glossary  improved 

A  work  environment  material  added  to  OPD  and  IPM 

A  IPPD  material  simplified  and  consolidated 

A  process  deployment  strengthened  in  IPM  and  OPF 
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A  History  of  Process 

Detailed  Command  vs.  Mission  Command 
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1850’s 

Train  Wreck  Management 


October  5,  1841  -  First  serious  train  wreck  in  the  US. 


Question:  How  to  run  a  large,  dispersed  organization? 


Six  Principles  of  Administration:* 

1.  Proper  division  of  responsibilities 

2.  Sufficient  power  conferred  to  enable 
same  to  be  fully  carried  out 

The  means  of  knowing  whether  such 
responsibilities  are  faithfully  executed 

Great  promptness  in  reporting  all 
derelictions  of  duties 

Information  obtained  through  a  system 
of  daily  reports  and  checks 

Adoption  of  a  system  to  enable  the 
General  Superintendent  to  detect  errors 
immediately  and  point  out  the  delinquent 

*The  Leader’s  Handbook  by  Peter  Scholtes,  quoting 
The  Visible  Hand:  The  Managerial  Revolution  in 
American  Business  by  Alfred  D.  Chandler,  (1977) 
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1880’s 


Command  Intent 


Helmut  von  Moltke,  Chief  of  the  General  Staff 
Prussian  Army,  1857  -  1888  (30  years) 

“No  plan  of  operations  extends  with  any  degree  of  certainty 
beyond  the  first  encounter  with  the  main  enemy  force.” 

Auftragstaktik  ( literally ,  “mission  tactics ”) 

Delegation  of  decision-making  authority  to 
subordinate  commanders  within  the  context  of 
the  higher  commander’ s  intent. 

mi  i  ,  r  •  •  i  Helmut  von  Moltke 

The  heart  oj  mission  command:  i  soo  1 89i 

uThe  advantage  which  a  commander  thinks  he  can  attain  through 
continued  personal  intervention  is  largely  illusory.  By  engaging 
in  it  he  assumes  a  task  that  really  belongs  to  others ,  whose 
effectiveness  he  thus  destroys.  He  also  multiplies  his  own  tasks 
to  a  point  where  he  can  no  longer  fulfill  the  whole  of  them.  ” 


Mission  Command:  Command 
and  Control  of  Army  Forces 

Field  Manual  No.  6-0 
Department  of  the  Army 
Washington,  DC,  1 1  August  2003 
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1910’ s 

The  One  Best  Way 


Frederick  Winslow  Taylor 

Assumptions : 

S  Workers  will  do  as  little  as  possible 

S  Workers  do  not  care  about  quality 

S  Workers  are  not  smart  enough  to 
know  the  best  way  to  do  their  job 

Taylor’s  View  of  Efficiency: 

Efficiency  comes  from  “knowing 
exactly  what  you  want  men  to  do  and 
then  seeing  that  they  do  it  in 
the  best  and  cheapest  way.” 

Expert  defines  the  best  way  through 
breaking  down  the  job  into  parts  and 
finding  the  best  way  to  do  each  part 

Pay  workers  extra  for  following  the 
method  determined  by  the  experts 


See:  The  One  Best  Way:  Frederick  Winslow  Taylor 
and  the  Enigma  of  Efficiency,  Robert  Kanigel,  1999 
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1920’  s 

Industrial  Training 

Charles  R.  Allen  -  New  Bedford,  Massachusetts 
On-the  job  training 
V  By  a  master  at  the  job 

*  Second  class  trainers  produce  second  class  learners 

*  Experts  know  how  to  do  the  job 

❖  But  they  need  training  in  how  to  train 

Four  Step  Method 

*  Preparation,  Presentation,  Application,  Testing 

1917  -  War  Ships  were  needed 

Allen:  Training  for  shipbuilders 

*  1 ,000  supervisors  trained  how  to  train 

*  88,000  shipbuilders  trained 

Wrote  “The  Instructor,  the  Man  and  the  Job” 


THE  INSTRUCTOR 
THE  MAN  AND  THE  JOB 

A  HAND  BOOK  FOR  INSTRUCTORS 
OF  INDUSTRIAL  AND  VOCATIONAL 
SUBJECTS 


BY  * 

CHARLES  R.  ALLEN 

accrues  agent  roR  industrial  training  or  bots  and  men,  Massa¬ 
chusetts  BOARD  or  EDUCATION,  AND  SUPERINTENDENT  or  INSTRUCTOR 
TRAINING,  U.  S.  S.  B.  EMERGENCE  FLEET  CORPORATION 
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1 930  3  s 

Unit  Command 

Truppenfuhrung  {“Unit  Command”) 

German  Army  field  manual,  1933/1934 

Section  4:  Lessons  in  the  art  of  war  cannot  be  exhaustively  compiled  in  the  form 
of  regulations.  The  principles  must  be  applied  in  accordance  with  the  situation. 
Simple  actions,  logically  carried,  out  will  lead  most  surely  to  the  objective. 

Section  6:  The  command  of  an  army  and  its  subordinate  units  requires  leaders 
capable  of  judgment,  with  clear  vision  and  foresight,  and  the  ability  to  make 
independent  and  decisive  decisions. 

Section  7:  An  officer  is  in  every  sense  a  teacher  and  a  leader. 

Section  10:  The  decisive  factor,  despite  technology  and  weaponry,  is  the  value  of 
the  individual  soldier.  The  battlefield  requires  soldiers  who  can  think  and  act 
independently ,  who  can  make  calculated,  decisive  and  daring  use  of  every  situation 
and  who  understand  that  victory  depends  on  each  individual. 
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1940’s 

Wartime  Production 

Wartime  Production 

Training  within  Industry  (TWI) 

*  Train  first  line  supervisors 

❖  Job  Instruction  -  how  to  train  workers 

❖  Job  Methods  -  how  to  improve  the  way  work  is  done 

❖  Job  Relations  -  how  to  treat  workers  with  respect 

Statistical  Process  Control  (SPC) 

*  W.  Edwards  Deming 

❖  Taught  defense  contractor  engineers  &  technicians 

❖  Over  30,000  trained 

*  Widely  used  in  defense  production 

TWI  &  SPC  were  ignored  by  manufacturers  after  1945. 
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1950’s 

TWI  &  SPC  move  to  Japan 


TWI  was  introduced  to  Japan  in 1947-48.  Toyota  adopted 
Job  Instruction  (JI)  in  1951,  Methods  (JM)  1952,  Job 
Relations  (JR)  in  1953.  JI  widely  used  to  this  day. 

JM  did  not  drive  deep  enough  into  Kaizen  and  the 
elimination  of  waste  strong  enough  to  suit  Taiichi 
Ohno,  so  he  had  Shigeo  Shingo  to  replace  it  with  a 
course  on  industrial  engineering  and  productivity. 


TRAINING 
WITHIN  INDUSTRY 
MATERIALS 


1915 


W.  Edwards  Denting 
System  of  Profound  Knowledge 

Appreciation  for  the  system 

Knowledge  of  Variation. 

Theory  of  Knowledge 

Psychology 
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1 960  ’s 

Toyota  Production  System 


Taiichi  Ohno 

Just-in-Time  Flow 

x  Eliminate  Waste 

S  Stop-the-Line  Culture  I 

x  Mistake-Proof  the  System 

Relentless  Improvement 

x  Learn  Through  Experimentation 


Taiichi 

Ohno 


rtipcru  nrauenoH  iruin 


Taiichi  Ohno 
(1912-1990) 


iBooks: 

The  Toyota  Production  System,  1978  (1988) 
Workplace  Management,  1982  (2007) 


Taiichi  Ohno's 

Workplace 

Management 
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Taiichi  Ohno 
Standard  Work 

There  is  something  called  standard  work,  but  standards 
should  be  changed  constantly.  Instead,  if  you  think  of  the 
standard  as  the  best  you  can  do,  it’ s  all  over.  The  standard 
work  is  only  a  baseline  for  doing  further  kaizen.  It  is  kai-aku 
[change  for  the  worse]  if  things  get  worse  than  now,  and  it  is 
kaizen  [change  for  the  better]  if  things  get  better  than 
now.  Standards  are  set  arbitrarily  by  humans,  so  how  can  they 
not  change? 

You  should  not  create  these  away  from  the  job.  See 
what  is  happening  on  the  gemba  and  write  it  down. 

From  Workplace  Management ,  by  Taiichi  Ohno,  originally  published  in  1982, 
from  translation  by  Jon  Miller,  Gemba  Press,  2007. 
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Taiichi  Ohno 
Motivation 

When  creating  Standard  Work,  it  will  be  difficult  to  establish 
a  standard  if  you  are  trying  to  achieve  ‘the  best  way.’  This  is  a  big 
mistake.  Document  exactly  what  you  are  doing  now.  If  you  make  it 
better  than  it  is  now,  it  is  kaizen.  If  not,  and  you  establish  the  best 
possible  way,  the  motivation  for  kaizen  will  be  gone. 

That  is  why  one  way  of  motivating  people  to  do  kaizen  is  to 
create  a  poor  standard.  But  don’t  make  it  too  bad.  Without  some 
standard,  you  can’t  say  ‘We  made  it  better’  because  there  is  nothing 
to  compare  it  to,  so  you  must  create  a  standard  for  comparison.  Take 
that  standard,  and  if  the  work  is  not  easy  to  perform,  give  many 
suggestions  and  do  kaizen. 

From  Workplace  Management ,  by  Taiichi  Ohno,  originally  published  in  1982, 
from  translation  by  Jon  Miller,  Gemba  Press,  2007. 


11 


November  07 


Copyright©2007  Poppendieck.LLC 


Taiichi  Ohno 
Ownership 

We  need  to  use  the  words  ‘you  made’  as  in  ‘follow  the 
decisions  you  made.’  When  we  say  ‘they  were  made’  people  feel 
like  it  was  forced  upon  them.  When  a  decision  is  made,  we  need  to 
ask  who  made  the  decision.  Since  you  also  have  the  authority  to 
decide,  if  you  decide,  you  must  at  least  follow  your  decision,  and 
then  this  will  not  be  forced  upon  you  at  all. 

But  in  the  beginning,  you  must  perform  the  Standard  Work, 
and  as  you  do,  you  should  find  things  you  don't  like,  and  you  will 
think  of  one  kaizen  idea  after  another.  Then  you  should  implement 
these  ideas  right  away,  and  make  this  the  new  standard. 

From  Workplace  Management ,  by  Taiichi  Ohno,  originally  published  in  1982, 
from  translation  by  Jon  Miller,  Gemba  Press,  2007. 
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Taiichi  Ohno 
Changing  Standards 

Years  ago,  I  made  them  hang  the  standard  work  documents 
on  the  shop  floor.  After  a  year  I  said  to  a  team  leader,  ‘The  color  of 
the  paper  has  changed,  which  means  you  have  been  doing  it  the 
same  way,  so  you  have  been  a  salary  thief  for  the  last  year.’  I  said 
‘What  do  you  come  to  work  to  do  each  day?  If  you  are  observing 
every  day  you  ought  to  be  finding  things  you  don't  like,  and 
rewriting  the  standard  immediately.  Even  if  the  document  hanging 
there  is  from  last  month,  this  is  wrong.’ 

At  Toyota  in  the  beginning  we  had  the  team  leaders  write 
down  the  dates  on  the  standard  work  sheets  when  they  hung  them. 
This  gave  me  a  good  reason  to  scold  the  team  leaders,  saying  ‘Have 
you  been  goofing  off  all  month?’  If  it  takes  one  or  two  months  to 
create  these  documents,  this  is  nonsense. 
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wm 


1970’s 

Theory  X  -  Theory  Z 


Theory  X 

1.  Most  people  dislike  work  and  don’t 
give  their  best  efforts  at  their  job. 

2.  Therefore  people  must  be  encouraged 
with  financial  incentives  or  threats  to 
work  towards  organizational  objectives. 

3.  Generally  people  would  rather  avoid 
responsibility  and  prefer  to  be  directed. 

Theory  Z 

People  are  motivated  by: 

1.  The  satisfaction  of  a  job  well  done 

2.  The  enjoyment  of  cooperating  with 
others  and  being  recognized  by  them 

3.  The  satisfaction  of  using  one’s 
abilities  to  the  fullest 


Kaoru  Ishikawa  TOTAL 

QUALITY 

CONTMH.T 

The  Jnndcirncntcil  principle  oj 
successful  management  is  to 
allow  subordinates  to  make 
full  use  of  their  ability. 

Everyone  who  is  connected  with  the 
company  ...  must  be  able  feel 
comfortable  and  happy  with  the 
company,  and  to  make  use  of  his 
capabilities  and  realize  his  potential. 

Top  managers  and  middle  managers 
must  be  bold  enough  to  delegate  as 
much  authority  as  possible.  That  is  the 
way  to  establish  respect  for  humanity  as 
your  management  philosophy. 


Theory  X  :  Douglas  McGregor 
Theory  Z:  Dr.  William  Ouchi 
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Mission  Command 
vs. 

Detailed  Command 
A  Comparison 


Mission  Command:  Command 
and  Control  of  Army  Forces 

Field  Manual  No.  6-0 
Department  of  the  Army 
Washington,  DC,  1 1  August  2003 


ovember  07 


1 

^  Mission  Command 

Detailed  Command^^ 

1  1 

*  Probabilistic 

•  Unpredictable 

Assumes  war  is 

*  Deterministic 

•  Predictable 

•  Disorder 

*  Uncertainty 

Accepts 

•  Order 

*  Certainty 

•  Decentralization 

•  Spontaneity 

•  Informality 

•  Loose  rein 

•  Self-discipline 

•  Initiative 

•  Cooperation 

•  Acceptable  decisions 
faster 

•  Ability  all  echelons 

•  Higher  tempo 

Tends  to  lead  to 

*  Centralization 

*  Coercion 

*  Formality 

*  Tight  rein 

*  Imposed  discipline 

*  Obedience 

*  Compliance 

*  Optimal  decisions, 
but  later 

*  Ability  focused  at  the 
top 

•  Implicit 

^«\/ertical  and  horizontal 

*  Tittecactive 

Communication 
types  used 

•  Explicit 

•  Vertical 

•  Linear 

*  Orgam£\. 

•  Ad  hoc 

Organization 
types  fostered 

*  Hierarchic 

•  Bureaucratic 

*  Delegating 

•  Transformational 

"^Leadership  styles 
encouraged 

•  Directing 

•  Transactional 

• 

Art  of  war 

S>ss. 

L 

Conduct  of  operations 

Appropriate  to 

^^’Technical/proceduraTN 

^•^.tasks 

1980’ s 

Total  Quality  Management 


1987:  ISO  9000  -  The  Generic  Quality  Standard _ 

f  What’s  Wrong  with  ISO  9000?  - \ 

S  ISO  9000  does  not  make  sure  you  are  doing  the  right  thing, 
only  that  you  are  doing  things  consistently. 

Certification  is  expensive  and  bureaucratic. 

Constant  improvement  of  processes  may  be  discouraged 
because  it  might  require  recertification. 


ISO  displaces  internal  motivation  to  do  good  work  with 
external  motivation  to  pass  the  inspection. 


Leaders 

Handbook 

S  -au  (E-E  rz-  I  I  1*14  Hu 
TSUH  PE4PLE  * 

■■HPUHfl  THE  E-E  ! LT 

Peter  R.  Scholtes 

ii-*— 


\ 

From  The  Leader’s  Handbook 

by  Peter  R.  Scholtes,  1998 
>-  J 


ISO  9000  certification  fits  the  mental  model  of  Direct  Command: 


*  Deterministic 

*  Predictable 
x  Order 

*  Certainty 

*  Centralization 

*  Coercion 

*  Formality 

*  Tight  rein 


*  Imposed  discipline 

*  Obedience 

*  Compliance 

*  Optimal  decisions,  but  later 

*  Ability  focused  at  the  top 

*  Explicit  Communication 

*  Hierarchic 

*  Bureaucratic 


Discipline  is  essential. 
Learning  is  fundamental. 
Documentation  is  incidental. 
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1990’  s 

CMM  &  CMMI 


Point: 

Disciplined  organizations  do  a  better  job. 

CMM  and  CMMI  present  a  good  set  of  process  areas  to  consider. 


Counter  Point: 


Kaizen  (Relentless  Improvement)  is  the  1st  -  not  the  4th  -  step. 

*  The  purpose  of  standards  is  to  provide  a  baseline  for  change. 
Building-Quality-In  is  not  optional. 

*  Finding  defects  during  final  verification  indicates  a  defective  process. 
You  can  have  it  all:  high  speed,  high  quality,  and  low  cost. 

*  The  maturity  of  an  organization  is  measured  by  the  speed  at 
which  it  reliably  and  repeatedly  executes  its  core  processes. 

Using  feedback  to  discover  value  is  better  than  following  a  plan. 

*  Plans  are  made  when  we  are  the  most  ignorant. 

KLOC’s  and  Function  Points  are  a  measure  of  complexity. 
x  The  fewer  lines  of  code  and  function  points,  the  better. 

Assessment  has  the  same  problems  as  ISO  9000. 


L 

E 

A 

N 
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High  Reliability  Organizations 


High  Reliability  Organizations 

X  Where  a  mistake  is  a  matter  of  life  and  death 
x  Firefighters 
x  Nuclear  Power  Plants 
x  Power  Grid  Dispatching  Centers 
x  Hospital  Emergency  Rooms 
x  Air  Traffic  Control 
x  Aircraft  Carriers 

High  Reliability  Organizations . . . 

Have  more  than  their  fair  share  of  unexpected  events 
Persistently  have  less  than  their  fair  share  of  accidents 


Common  Characteristic 

Mindfulness* 


*  See  Managing  the  Unexpected:  Assuring  High  Performance  in  an 
Age  of  Complexity .by  Karl  E.  Weickand  Kathleen  M.  Sutcliffe,  2001 
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Mindfulness 


Preoccupation  with  Failure 

Anticipate  and  become  aware  of  the  unexpected 

*  Anything  that  can  go  wrong  will  eventually  go  wrong 

Reluctance  to  Simplify 

Learn  to  live  in  a  complex,  unpredictable  world 

*  “Standard  Procedures”  cannot  replace  thinking  people 


Sensitivity  to  Operations 

Be  attentive  to  the  front  line  where  the  work  gets  done 


*  Go  “to  the  Gemba”  and  see  for  yourself 

Commitment  to  Resilience 

Learn  to  detect,  contain  and  bounce  back  from  failure 

*  STOP  -  Investigate  -  Find  Root  Cause  -  Rectify 

Deference  to  Expertise 

Move  decisions  to  the  front  line 

*  Use  Mission  Command,  not  Detailed  Command 


From  Managing  the  Unexpected:  Assuring  High  Performance  in 
an  Age  of  Complexity  by  Karl  E.  Weickand  Kathleen  M.  Sutcliffe 


2000  ’s 
Six  Sigma 


Counter  Point: 

Focus  on  Variation 


Point: 

Excellent  tool  set  for  using  the  Scientific  Method 
Broad  education  in  toolset  throughout  the  organization 
V  Defines  Quality  as  the  “Voice  of  the  Customer”  (VOC) 
Closer  to  mission  control  than  detailed  control 


*  Variation  comes  in  two  forms:  Assignable  Cause  &  Chance  Cause 


❖  Trying  to  eliminate  “chance  cause”  usually  makes  the  situation  worse 


*  In  a  development  process,  variation  is  desirable  -  it  creates  knowledge. 


Focus  on  VOC 

*  VOC  drives  sustaining  technologies,  but  not  disruptive  technologies 

“The  truth  is  that  no  system  seems  all  that  good  at  picking  winners  in  advance.  What 
makes  a  system  successful  is  its  ability  to  generate  lots  of  losers  and  then  to  recognize 
them  as  such  and  kill  them  off.  ”  -  James  Surowiecki,  The  Wisdom  of  Crowds 

“Nothing  will  kill  innovation  faster  than  trying  to  manage  it,  predict  it, 

and  put  it  on  a  timeline.  ”  -  Vishva  Dixit,  vice  president  for  research  of  Genentech 
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Plank  Road  Fever 


Information  Cascade 

“The  first  plank  roads  were  a  huge  success. 
People  looking  for  a  solution  to  the  road 
problem  found  one  ready-made  at  hand. 

As  more  people  built  plank  roads,  their 
legitimacy  became  more  entrenched  and  the 
desire  to  consider  alternate  solutions  shrank. 
It  was  years  before  the  fundamental  weakness 
of  plank  roads  -  they  didn ’t  last  long  enough 
-  became  obvious.  ” 

James  Surowiecki,  The  Wisdom  of  Crowds 


USA:  Late  1840’s  -  mid  1850’s 

Massive  boom  in  plank  road  construction 

^High  capital  investment 

^Numerous  large  and  small  investors 
>^To  be  paid  for  with  tolls 
Immediate,  positive  results 

^Far  superior  to  muddy,  rutted  roads 
Dramatic  decrease  in  travel  time 
Expanded  rural  markets 

BUT 

Roads  deteriorated  in  4  years 
*Half  the  projected  lifespan 
Maintenance  costs  were  very  high 
*  Annual  costs  were  20-30%  of  initial  cost 
Most  plank  roads  were  soon  abandoned 


Of  58  large  companies  that  have  announced  Six  Sigma 
programs,  91  percent  have  trailed  the  S&P  500  since. 

Fortune,  July  2006 
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Thank  Y  ou ! 

More  Information:  www.poppendieck.com 


mary@poppendieck.com 


Mary  Poppendieck 


www.poppendieck.com 


Complete 


Your  complimentary 
use  period  has  ended. 
Thank  you  for  using 
PDF  Complete. 


CMMI: 

Fitting  a  Vision  to  Program  Execution  Needs 

November  13,  2007 


Mark  D.  Schaeffer 

Director,  Systems  and  Software  Engineering 
Office  of  the  Under  Secretary  of  Defense  (AT&L) 


t  Complete 


Your  complimentary 
use  period  has  ended. 
Thank  you  for  using 
PDF  Complete. 

\nded  Features 


CMMI  Vision* 


The  initial  vision  for  CMMI  was  to  integrate  the 
competing  maturity  models  and  provide  a 
framework  for  more  consistent  process 

improvement 

A  Cause  integration  of  the  functional  disciplines  within 
organizations  and  across  programs 

Alncrease  systems  engineering  and  software  process 
maturity  as  organizations  migrate  from  the  sun-setting 
CMMs  to  CMMI 

Build  on  and  improve  the  significant  work  done  by 
many  to  establish  best  practices 


*  Extract:  2004,  2005,  2006  CMMI  Conference  Keynotes 


1  Complete 


use  period  has  ended. 
Thank  you  for  using 


Your  complimentary 


PDF  Complete. 


ed  Features 


AWe  have  attained  the  original  vision 
A  We  have  taken  steps  to  address  CMMI  issues: 

i  Integrity  issues  with  appraisals 
T  Guidance  for  acquiring  organizations 

ACurrent  Issues: 

T  Staged  vs.  Continuous 

A  Cost  of  levels  versus  Return  on  Investment 
T  High  Maturity 

A  Level  4  and  5  inconsistency 
A  High  maturity  appraisals  and  training 

A  Relationship  to  other  continuous  process  improvement  initiatives 

T  Next  Gen  Process  Improvement 

A  How  do  we  revise  the  CMMI  vision  to  meet  program  execution 
needs? 


A  PDF 

t  Complete 


Your  complimentary 
use  period  has  ended. 
Thank  you  for  using 
PDF  Complete. 


inded  Features 


pessary  but  not  sufficient. 


A  We  have  revitalized  Systems  Engineering  Policy,  Guidance, 
Education  and  Traininge 

A  We  have  driven  good  systems  engineering  practices  back 
into  the  way  the  acquisition  community  does  business,  and 
have  had  a  positive  impact  on  programse 

A  We  have  expanded  the  boundaries  to  include  increasingly 
important  enablers  for  sound  SE  application^ 

A  We  have  a  rigorous  process  to  capture  what  went  wrong... 

Ae  but  failed  to  change,  root  cause  behavior  that  leads  to 
programs  that  do  not  meet  cost,  schedule,  and  performance 
expectations^  adequate  maturity  at  program  initiation 

What  are  the  systemic  issues  that  need  to  be  addressed? 


1  Complete 


use  period  has  ended. 
Thank  you  for  using 


Your  complimentary 


ed  Features 


PDF  Complete.  phe  Real  World  and  CMMI: 

:eafures  liship  between  CMMI  and  Program 


* 


A  Programs  adhering  to  organizational  processes: 

T  85%  of  programs  find  the  supplier  performs  their  defined  processes 
with  minor  non-compliance 

A  For  programs  that  dond>  adhere  to  processes: 

T  Primary  reasons  are  schedule,  cost,  and  customer  impact 

A  There  does  not  appear  to  be  a  link  between  maturity  levels 
and  program  performance 

V  No  correlation  between  maturity  levels  and  cost  variance  or  CPI 

V  Indication  of  negative  correlation  between  ML  and  schedule  variance 
orSPI 

A  There  does  not  appear  to  be  cost  and  schedule 
improvement  from  ML3  ->  ML5 


*DCMA  Data  Findings  (2007)  5 


t  Complete 


Your  complimentary 
use  period  has  ended. 
Thank  you  for  using 
PDF  Complete. 

\nded  Features 


m  Support  Review  Activity 

(since  March  2004) 


A  PSRs/NARs  completed:  42 
A  AOTRs  completed:  10 
A  Nunn-McCurdy  Certifications:  10 
A  Participation  on  Service-led  IRTs:  2 
A  Technical  Reviews:  9 
A  Reviews  planned  for  remainder  FY07 

□  PSRs/NARs:  12 

□  AOTRs:  2 

□  Nunn-McCurdy:  6 

□  Technical  Reviews:  3 


Decision  Support  Reviews 


Service-Managed  Acquisitions 


Programs  by  Domain  Area 


Air  Force 
39% 


Marine 
Corps  8% 


Agencies 

8% 


Army 

26% 


Missiles  8% 


Fixed  Wing 
21% 


Business  3% 
Space  5% 


Rotary  Wing 
16% 


Other  7% 


Munitions  4%  Ships  7% 


Land  15% 


C2-ISR  10% 
Unmanned  4% 


Navy 

19% 
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PDF  Complete. 


and  Expanded  Features 


0  Emerging  Systemic  Issues 

“Deep  Dive”  Program  Reviews  since  Mar  04) 


1.  Management 

2.  Requirements 

3.  Systems  Engineering 

4.  Staffing 

5.  Reliability 

6.  Acquisition  Strategy 

7.  Schedule 

8.  Test  Planning 

9.  Software 

10.  Maintainability/Logistics 


AlPT  roles,  responsibilities,  authority,  poor  communication 
A  Inexperienced  staff,  lack  of  technical  expertise 

A  Creep/stability 

ATangible,  measurable,  testable 
A  Lack  of  a  rigorous  approach,  technical  expertise 

A  Process  compliance  ^  Program  execution 

A  Inadequate  Government  program  office  staff 

AAmbitious  growth  curves,  unrealistic  requirements 
A  Inadequate  rtest  timeofor  statistical  calculations 

A  Competing  budget  priorities,  schedule-driven 
A  Contracting  issues,  poor  technical  assumptions 

A  Realism,  compression 
A  Breadth,  depth,  resources 

A  Architecture,  design/development  discipline 
A  Staffing/skill  levels,  organizational  competency  (process) 

A  Sustainment  costs  not  fully  considered  (short-sighted) 
ASupportability  considerations  traded 


1  Complete 


use  period  has  ended. 
Thank  you  for  using 


Your  complimentary 


PDF  Complete. 


ed  Features 


Junn-McCurdy  Breaches 


AlMine  key  visible  failures: 

T  Change  in  doctrine,  driving  quantity  or  mission  changes 
i  Requirements  problems  (immature,  unrealistic,  not  stable,  creep,  etc) 
i  Lack  of  a  robust  baseline 

i  Inadequate  SE/T&E,  risk  management,  and/or  FMECA 
T  Inadequate  staffing/experience/oversight  levels 
T  Poor  reliability 
T  Acquisition  reform 

T  Schedule/cost  realism  (concurrency,  estimation,  etc) 

T  Contract  (warranty,  price  curves,  TSPR,  etc) 


Processes  in  place  £  Program  Execution 
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t  Complete 
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use  period  has  ended. 
Thank  you  for  using 
PDF  Complete. 

\nded  Features 


10  Emerging  Systemic  Issues 
from  Triage  Assessment 


1.  Insufficient  trade  space  (resources) 

24  programs  (37%) 

2.  Insufficient  schedule  trade  space 

22  programs  (34%) 

3.  Budget  not  properly  phased/  magnitude  to  support  planned 
developmental  (SE,  T&E,  production,  etc.)  efforts 

17  programs  (26%) 

4.  Concurrent  test  program 

16  programs  (25%) 

5.  Insufficient  performance/  requirements  trade  space 

16  programs  (25%) 

6.  Operational  or  Developmental  performance  results  indicate 
not  effective/  suitable  or  KPPs  not  meeting  threshold 

15  programs  (23%) 

7.  Lack  of  JROC-validated  requirements  document  for  basic 
program  (ORD,  CDD,  CPD) 

14  programs  (22%) 

8.  Funding  instability 

14  programs  (22%) 

9.  Inadequate  implementation  of  EVMS  and  use  of  EVM  as  a 
vehicle  for  planning,  executing,  and  controlling  the  program 

14  programs  (22%) 

10.  Current  unit  cost  factors  indicate  significant/  critical  APB 
breach 

12  programs  (19%) 
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\nded  Features 


Root  Cause  Analysis 
Emerging  Results 


A  Emerging  systemic  analyses  point  to  the  following  2  core  root  cause 
areas  and  their  top  4-5  drivers: 


An  “ Execution  Discipline” problem...  Solutions  need  to  address 
“state-of-the-practice”  vice  “state-of-the-art” 
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ed  Features 


Let’s  Review... 


Astaged  vs.  Continuous 

T  DoD  does  not  encourage  use  of  levels 
T  Current  practice  of  attaining  levels 

A  Continues  to  drive  program/enterprise  cost 
A  Does  not  correlate  to  program  success 

A  Contributes  to  acquirers  and  developers  not  having  to  rthinkoabout 


program  execution 


AHigh  Maturity 

V  High  maturity  is  ill  defined/narrowly  applied  --  vice  adopting  CPI  in  all 
required  areas 

ANext  Gen  Process  Improvement 

V  Starting  programs  right  V  disciplined  execution  V  highest  probability 
for  program  success 

T  We  must  address  state  of  the  practice,  vice  state  of  the  art 


Fit  CMMI  Vision  to  Pro/gram  Execution  Needs 
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VIMI  Vision  for  the  Future 


ACurrent 

i  5  Levels 

T  Three  SCAMPIs 

T  Constellations  for  major 
stakeholders 

T  High  Maturity  improvement 
plans 

T  Cost  of  integrity 


A  Future 

T  Foundational  best  practices 

T  Tailored  to  organizational, 
domain,  and  program  needs 

T  Focus  areas  to  extend  the 
foundation  to  specific  interest 
areas  (e.g.  safety,  COTS) 

T  Structured  measurement 
process  T  aligned  with  tools 
for  high  maturity 


CMMI  should  continue  to  ensure  foundational  best 
practices;  tailored  to  Org/Domain/Program  needs 
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Backup 
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.  Survey  and  Data  Collection 


A  Survey  conducted  in  response  to  OUSD  (AT&L)  request: 

A  ris  there  a  relationship  between  CMMI  levels  and  program 
performance?© 


A  85-142  programs  reported  each  quarter 


A  ACAT  Levels  reported 

A  ACAT  IAC  T  9  programs 
A  ACAT  1AM  T  5  programs 
A  ACAT  1C  T  33  programs 
A  ACAT  ID  T  79  programs 
A  ACAT  NT  16  programs 


A  Claimed  maturity  levels  (MLs) 

A  l\/ll  A  V  Q  nrnnromo 


A  ML  1  ' 
A  ML2‘ 
A  ML3‘ 
A  ML4‘ 


3  programs 
1  program 
47  programs 
17  programs 


A  ML  5  T  74  programs 


*  Excerpt  from  DCMA  Data 
Call  Results  briefing  -  Nov  07 


#  of  Programs 
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\  Study  -  Data  Assumptions 


A  Many  survey  questions  are  subjective  (Local  DCMA  viewpoints) 

A  If  both  Maturity  Level  (ML)  and  Capability  Level  (CL)  reported,  only 
captured  ML 

A  Only  captured  highest  Maturity  Level  achieved 

A  Example:  ML  5  SW  only  with  ML  3  for  SE;  ML  5  data  was  used 

A  If  a  range  (eg.  5-10%)  was  given  for  any  EV  data,  highest  value 
(10%)  was  recorded 

A  Only  used  latest  PO/contract  for  a  program 

A  Not  all  the  totals  will  add  up  to  the  sample  size  due  to  unanswered 
questions 

A  Have  deleted  some  EV  data  points  due  to  suspect  data  (Suspect 
decimal  point  issues) 
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"When  the  Only  Tool  You  Have 
is  a  Hammer, 

Every  Problem  Begins  to  Look 
Like  a  Nail" 


Sam  Fogle 
ACE  Guides,  LLC 
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•  Tools?  You  mean  like  a  hammer? 

•  If  it  ain't  broke,  don't  fix  it.  But  if  it 
IS,  you'll  probably  need  a  tool. 

•  I  watched  that  danged  exercise  video 
15  times  and  I  ain't  lost  a  pound. 
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•  Work:  something  on  which  exertion  or 
labor  is  expended;  a  task  or  undertaking 


•  Tool:  anything  used  as  a  means  of 
accomplishing  a  task  or  purpose 

Tools  help  us  accomplish  our  work 

What  tools  does  your  organization  use? 


*Definitions  from  Dictionary.com 
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Is  do  you  need? 


Does  your  organization  have  a  good  set  of 
tools  to  do  its  work? 


Well,  we  get  thelfork  done  so  I  guess  so 


We  don't  know 
would  help,  so 


ny  other  tools  that 
ess  so 


•  We  do  a  reasonable  job  of  keeping  up 
with  what  is  available,  and  investigating 
new  tools,  so  I  guess  so 
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it  for  us? 


What  does  moving  up  a  CMMI  level  get  you? 

•  A  nice  certificate  to  hang  on  the  wall 

•  Bragging  rights 

•  Ability  to  bid  on  some  new  work 

•  NEW  TOOLS 
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But  if  it  IS, 

you'll  probably  need  a  tool. 
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We  are  in  a  Level  1  organization  and  we  hear 
these  type  of  complaints: 

•  "Our  project  managers  never  seem  to  be 
able  to  live  within  the  budgets  we  give 
them." 


Project  Planning  -  estimating 
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We  are  in  a  Level  1  organization  and  we  hear 
these  type  of  complaints: 

•  "Projects  seems  to  be  on  track  until  the  last 
minute  (90%  complete!)  and  then  there  is 
a  train  wreck." 


Project  Planning  -  documenting  plans 
Measurement  and  Analysis  -  analyzing  measures 
Project  Monitoring  and  Control  -  monitoring  progress 
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Our  organization  has  moved  up  to  Level  2,  but 
now  we  hear  these  type  of  complaints: 

•  "Every  project  has  its  own  custom 
processes,  but  new  projects  don't  know 
which  ones  to  adopt,  or  whether  to  create 
their  own." 


Organizational  Process  Definition  -  developing 
standard  processes,  tailoring  guidelines 
Integrated  Project  Management  -  defining  project's 
process 
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Our  organization  has  moved  up  to  Level  2,  but 
now  we  hear  these  type  of  complaints: 

•  "It  seems  that  for  every  project  that  gets 
into  trouble,  we  can  look  back  to  one  or 
two  critical  choices  that  were  made  too 
hastily." 


Decision  Analysis  and  Resolution 
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Our  organization  has  moved  up  to  Level  2,  but 
now  we  hear  these  type  of  complaints: 

•  "It  seems  that  a  lot  of  the  issues  we 

discover  in  testing  are  simple  mistakes  that 
the  author  couldn't  see  because  they  were 
too  close  to  the  work." 


Verification  -  Peer  Reviews 
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We  are  now  proudly  Level  3,  but  people  are 
still  complaining: 

•  "We  keep  'improving'  our  processes,  but 
how  do  we  know  if  they  are  good  enough." 


Organizational  Process  Performance  -  establishing  a 
quantitative  understanding  of  process  performance 
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We  are  now  proudly  Level  3,  but  people  are 
still  complaining: 

•  "Our  PMs  do  a  great  job  of  planning  and 
tracking  their  projects  and  integrating  their 
plans,  but  even  if  things  are  on  plan  today, 
they  still  worry  whether  they  will  be  able  to 
meet  their  project's  end  goals." 


Quantitative  Project  Management 
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Level  4  is  great,  but  do  you  think  that  shut 
them  up?: 

•  "We  understand  our  processes  and  what  we 
can  expect  from  them,  but  it  seems  that  we 
keep  having  to  fix  the  same  type  of 
problem  time-after-time." 


Causal  Analysis  and  Resolution  -  stopping  defects 
before  they  occur 
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Level  4  is  great,  but  do  you  think  that  shut 
them  up?: 

•  "Our  processes  are  very  stable.  In  fact 
they  are  so  stable  they  haven't  changed  in 
years.  I  can't  believe  that  there  aren't 
ways  out  there  to  do  some  of  the  things  we 
do." 


Organizational  Innovation  and  Deployment  - 
incremental  and  innovative  improvements 
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Get  the  Rating 

Cost 

X 

Benefits 

•Certificate 
•Bragging 
•Able  to  bid 

Improve  the 
Organization 

X+A 

•Capability 

(tools) 

•Above  plus 

•Improved 

performance 

o«< 


plete 


Your  complimentary 
use  period  has  ended. 
Thank  you  for  using 
PDF  Complete. 
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"Investigation,  Measures  and  Lessons  Learned  about  the 

Relationship  between  CMMI  Process  Capability  and  Project 
and  Program  Performance." 

Process  Capability  =  you  have  demonstrated  that  you  have  and 
can  use  the  tools  that  characterize  a  level 


Project  and  Program  Performance  =  You  have  adopted  and 
effectively  use  those  tools 


Performance 


Capability 
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RANGE  SAFETY 

oa/s  are  similar  to  CMMI-DEV  RSKM  process  area 


Range  Safety  manages  risk  in  support  of  different  product  lines.  Each  product  line  presents 
different  range  hazards  and  requires  facilities,  equipment,  and  skills  to  ensure  safe  operation. 


SPECIFIC  GOAL  1:  Prepare  for  risk  management. 
Establish  policies  and  procedures  at  this  level... 


Mission 

Areas 


Weapons 

Separation 


Supersonic 

Operations 


Targeting 

Lasers 


Airborne  and 
Surface 
Gunfire 


Tactical 

Missiles 


UAVs 


Test  Plans 


Each  mission  area  has  multiple 
test  plans  associated  with  it. 

SPECIFIC  GOAL  2:  Identify  and 
analyze  risks  at  this  level 


Each  test  plan  has  multiple  test 
events. 


Test  Events 


SPECIFIC  GOAL  3:  Mitigate  risks. 
Develop  and  implementing  risk 
mitigation  plans  at  this  level. 
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Range  Safety  be  improved? 


A  What  is  our  current  performance? 

i  Can  we  provide  management  with  a  rmetricd? 
T  What  is  our  desired  performance? 

T  How  can  we  excel? 


A  What  is  “improvement”? 

T  Improving  effectiveness? 

T  Improving  efficiency? 

T  Can  we  do  both? 


A  What  should  our  strategy  for  improvement  be? 

T  Lean  /  Six  Sigma  ? 

T  CMMI  ? 
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1MI  to  visualize  improvement? 


Our  Questions:  Will  a  CMMI  improvement  effort  be  useful? 

AWill  the  benefits  be  clear? 

Awill  the  benefit  in  effectiveness  or  efficiency  justify  the  depth,  breadth,  time,  and 
cost  of  the  CMMI  improvement  effort? 


Problem:  To  the  novice,  the  benefit  of  implementing  CMMI  is 
offset  by  its  intimidating  complexity. 

AThe  Intro  to  CMMI  course  is  a  sip  out  of  a  very  large  fire  hydrant. 

AThe  sheer  volume  of  information  delivered  in  three  days  is  overwhelming,  intimidating,  and 
discouraging  to  students. 

AHow  does  CMMI  help? 

AHow  can  we  use  the  CMMI  tool  to  measure  and  improve  our  current  performance? 

Solution:  The  Capability  Waypoint  Matrix  tool  can  show  at 
a  glance: 

ACapability  levels  of  specific  practices 
AEffectiveness  of  individual  process  areas. 

AEfficiency  of  individual  process  areas 

ASimple  but  detailed  insight  into  the  improvement  status  of  the  entire  organization. 
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ypoint  Concept  of  Improvement 


flncrease  effectiveness,  efficiency,  or  botho 


A  Effectiveness 

i‘  Meet  process  area  specific  goals,  if. . . 

i‘  Specific  goals  are  traceable  to  organization ’s  mission 
objectives 


A  Efficiency: 

i  Meet  goals  faster,  cheaper,  better 

i‘  Increasing  CMMI  capability  levels  implies 
improvement  in  schedule,  cost,  and  quality 
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veness  and  Efficiency 


Range  Safety  example 


A  Range  Safety  Effectiveness: 

Meet  organization’s  goals 

i  Technical  competency  &  proficiency  in  safety  support  of  weapons  tests 
A  “Prepare  for  risk  management’ 

i  Conduct  weapons  tests  without  unacceptable  risk  to  personnel  or 
property 

A  “Identify  and  analyze  risks” 

A  “Develop  &  implement  risk  mitigation  plans” 

A  Range  Safety  Efficiency: 

Meet  goals  faster,  cheaper,  better,  smarter 

T  Minimize  costs  of  risk  management  without  compromising  safety 

T  CMMI  model  capability  levels  address  improvement  in  cost,  schedule 
and  performance 

A  Improvement:  Increase  effectiveness,  efficiency,  or  both 


Visualizing  Improvement  with  Capability  Waypoints 
Approved  for  public  release;  distribution  is  unlimited. 


6 


7th  Annual  CMMI  Technology  Conference 
November  12-15,  2007 


_"*PDF 

■»  Complete 


Your  complimentary 
use  period  has  ended. 
Thank  you  for  using 
PDF  Complete. 


» upgrade  to 


and  Expanded  Features 
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ty  Specific  Goals  &  Practices 


Effectiveness  means  “meeting  specific  goals” 


1 .  Prepare  for  Risk  Management 

Different  for  various  categories  of  test  events  (Bombs,  guns,  lasers,  UAVs,  etc) 

A  Determine  risk  sources  and  categories 

A  Define  risk  parameters 

A  Establish  a  risk  management  strategy 


2.  Identify  and  Analyze  Risks 

For  each  test  plan 

A  Identify  risks 

A  Evaluate,  categorize,  and  prioritize  risks 


3.  Mitigate  Risks 

For  each  test  event 

A  Develop  risk  mitigation  plans 

A  Implement  risk  mitigation  plans 

We  are  effective  if  we  are  proficient  in  all  range  safety  practices  and 
achieve  the  range  safety  specific  goals. 

We  are  ineffective  if  range  safety  specific  goals  are  not  met. 
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ty  Levels  as  a  Measure  of  Efficiency 

ability  levels  results  in  improved  quality  and  efficiency 


A  Capability  Level  1  “PERFORM” 

(i.e.,  “Be  Effective’’) 

T  Achieve  specific  goals  * 

A  Perform  base  practices 

A  Capability  Level  2  “MANAGE” 

(i.e.,  better,  smarter,  etc.) 

T  Commitment  to  perform  * 

A  Establish  organization  policy 
T  Ability  to  perform 
A  Plan  the  process 
A  Provide  resources 
A  Assign  responsibility 
A  Train  people 
T  Direct  implementation 
A  Manage  configurations 
A  Involve  stakeholders 
A  Monitor  &  control  process 
T  Verify  implementation 

A  Objectively  evaluate  adherence 
A  Review  status  with  higher  management 


A  Capability  Level  3  “DEFINE” 

T  Ability  to  perform 

A  Establish  a  defined  process 

T  Directing  implementation 

A  Collect  improvement  information 

A  Capability  Level  4  “QUANTIFY” 

T  Quantify  the  Process 

A  Establish  quantitative  objectives 
A  Stabilize  sub  process  performance 

A  Capability  Level  5  “OPTIMIZE 

T  Optimizing  Process 

A  Ensure  continuous  improvement 
A  Control  root  causes 


*  “Common  Features”  terminology  (commitment  to  perform,  ability  to  perform,  direct  implementation,  etc.)  , 
used  in  CMMI-DEV  version  1. 1  but  not  in  version  1.2,  are  retained  because  they  help  us  explain  the  value 
and  importance  of  generic  practices. 
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lalizing 


Improvement 


EFFICIENCY 

Meet  organization  goals 
faster,  better,  cheaper 


EFFECTIVENESS 

Completeness’  of  meeting 
organization’s  goals 


/ 

Goal  1 


Goal  2 

Goal  N 
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1958 

1963 

1975 


i  1996 
1999 
N  2000 

-  2001 

2003 

2003 

2006 

2007 


Robert  J.  Collier  Trophy  -  F-104  Starfighter 

Robert  J.  Collier  Trophy  -  SR-71  Blackbird 

Robert  J.  Collier  Trophy  -  F-16  Fighting  Falcon 

SW-CMM®  Maturity  Level  3 

SW-CMM®  Maturity  Level  4 

Shingo  Prize  -  Fort  Worth,  TX 

Robert  J.  Collier  Trophy  -  F-35  STOVL  lift  fan 

ISO  9001/AS  9100 

Shingo  Prize  -  Palmdale,  CA 

Robert  J.  Collier  Trophy  -  F-22  Raptor 

CMMI  Maturity  Level  3 


Utah  State  Saiversita 


Robert  J.  Collier  Trophy  is  a  national 
award  honoring  those  who  had  made 
significant  achievements  in  the 
advancement  of  aviation. 


®  CMM  is  registered  in  the  U.S.  Patent  and  Trademark 
Office  by  Carnegie  Mellon  University. 


The  Shingo  Prize  was  established  in 
1988  to  recognize  companies  that 
achieve  world-class  manufacturing 

status. 
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Organizational  strategy 

-  Reflect  anticipated  business  base 

-  Include  all  major  development  sites 

-  Touch  full  span  of  our  products 

•  From  JSF,  largest  aircraft  program  in  history 

•  To  smaller  projects  of  30-35  heads 


. * 


•  R&D  projects 

Result:  Chose  3  appraisal  projects 
-  F-35  -  Fort  Worth.  Texas 


RATTLRS- 
Revolutionary 
Approach  To  Time- 
critical  Long  Range 
Strike 


/  Projects  Strategically  Selected 
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Used  our  corporate  expertise 

-  Understood  model  and  addressed 
challenges 

-  On-site  participation  for  SCAMPI 


Adopted  best  practices 

-  LM  Continuous  Appraisal  Methodology 
(CAM) 

-  Lockheed  Martin  Integrated  Enterprise 
Process  (LM-IEP)  Architecture 

-  Best  practices  from  sister  companies 


Built  our  approach  using  the  best! 
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A  comprehensive  enterprise  architecture  serves  as  the 
foundation  for  continuous  process  improvement 


A.1  Organizational  Management 
A.2  Strategic  Planning 
A.3  Quality  Management 
A.4  Ethics  &  Business  Conduct 
A.5  Legal 

A.6  Communications 


B.1  Process  Management 
B.2  Work  Environment  Management 
B.3  Technology  Management 
B.4  Contracts 

B.5  Workforce  Management 
B.6  Finance 

B.7  Supplier  Agreements  &  Procurement 
B.8  Security 

B.9  Property  Management 


D.3.1  Stakeholders  Needs 
Analysis 
D.3.2  Requirements 
Development 

D.3.3  Architectural  Design 
D.3.4  Detailed  Design 
D.3.5  Implementation 
D.3.6  Integration 
D.3.7  Verification 
D.3.8  System  Validation 


Business  Execution  Processes 


Program  Execution  Processes 


rr 


Common  Manaaement  Processes 
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_ Continuous  Appraisal 

Method  (CAM)  Visits 


Revised  1/2007  Plan 


2006 

Is*  Q 

2nd  Q 

3rd  Q 

4th  Q 

Is*  Q 

♦  ♦♦♦ 


♦ 


Continuous  Appraisal 
Method  (CAM)  Visits 


OOOO 
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2007 

1st  Q  2nd  Q  ,  3rd  Q  4th  Q 


☆ 


SCAMPI  B  SCAMPI 

Go-No  Go  II  A 
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2nd  Q  3rd  Q  4th  Q 


SCAMPI  SCAMPI 
B  A 


Earlier  vl.1  sunset  schedule  required  accelerated  SCAMPISM 
preparation  to  protect  our  vl.1  OSP  investment 
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Readiness  SCAMPI 

Readiness 
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Coverage 
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Review 
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Review 
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Review 
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Review 

5/2-3 
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3/16 
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i 

i 
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4/13 
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i 

i 

; 

; 

; 
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i 

i 

i 

Finalize  and  Release 
Processes 

Finalize  and  Release  Program 
Plans  &  Procedure 


Used  our  project  management  strengths 
AClear  lines  of  authority 
ARisk/schedule  management 
AAccountability  reviews 


Conduct  &  Attend  Process  Training 

_  i 

i 

Implement  Plans/Procedures  &  Collect  Evidence 

Conduct  and  Support  Internal  Evidence  Evaluations 

i 

Improve  Processes  /  Update  OSP,  PDPs,  and  Other  Products 

Address 
Findings 


Features  of  the  LM  Continuous  Appraisal 
Method  (CAM)  incorporated  into  CMMI 
review  teams  and  independent  PHD  reviews 


Address 

Information 

Needs 
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Document  review 

-  Over  3900  items  examined 

-  Included  classified  data 

-  Only  9  Information  Needs 

-  No  major  findings 
Interviews 

-  Over  100  participants  at  3  sites 

-  Conducted  9  group  interviews 

-  Included  major  engineering 
disciplines 

•  Hardware 

•  Software 

•  System 

Appraisal  Team 

-  Team  consisted  of  11  members 


SYSTEMS  AND 
SOFTWARE 
CONSORTIUM 


The  Systems  and  Software  Consortium  completed  aCMMI®- 
Based  Appraisal  on  August  30,  2007  in  accordance  with  the 
Standard  CM  Ml*  Appraisal  Method  for  Process  Improvement 
(SCAMPI8*1},  VI  .  1  and  determined  that 

Lockheed  Martin  Aeronautics 

achieved 

Process  Maturity  Level  3 

as  defined  by  the  SEI  CMMI*  Version  1.1  SE/SW 


Continuous  Representation. 


G  ejie  Jorgensen,  SSCS 

SEt  Authorized  Lead  Apprais  er 


DfHW  Atfism,  SSC I 
Appraisal  Team  Member 


•  5  internal,  6  external 

•  4  SCAMPI  Lead  AppraisersSM 

-  Consistent  mini-team  assignments 
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Improved  Company  Behaviors 


•  Vice  President  assigned  full  time  to  sponsor  the  effort 

-  Management  commitment  from  the  company  president  down 

-  “Direct  leadership  from  the  sponsor  provided  solid  foundation  for 
process  improvement ” 

•  Feedback  from  programs  and  functions 

-  Programs  are  more  involved  in  process  changes  and  deployment  of 
processes 

•  Changes  are  pre-coordinated  with  programs 

•  Programs  incorporate  changes  prior  to  formal  deployment 

-  Increased  use  of  disciplined  processes 

-  Improvements  identified  on  one  program  are  shared  across  programs 
in  a  more  timely  manner 

-  Improved  communications  across  programs ,  sites  and  functions 

-  Program  integration  -  good  communication  and  consistent 
expectations 
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•  Longest  effort:  Development  &  release  of  the  CMMI  compliant  OSP 

-  OSP  “what’s  in  vs.  what’s  out”  was  a  big  issue 

-  Processes  were  written  within  each  functional  area 

-  Process  owners  previously  had  no  common  vision  /  framework 

-  Was  difficult  to  obtain  program  engagement  -  too  busy 

-  Few  subject  matter  experts  were  available 

•  It  pays  to  keep  it  simple 

-  Took  advantage  of  common  metrics  already  in  use 

-  Combined  peer  review  methods  into  one  common  process 

-  Aligned  auditing  groups  under  one  process 

-  Standardized  training  across  multiple  functions 
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-  Recognition  of  model  flexibility 

-  PHD  reviews  to  accelerate  evidence  collection 

-  Documented  our  standard  approach  for  generic  practices 

-  Structured  SCAMPI  opening  briefings  to  maximize  GP  coverage 
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Institutionalization  Measures:  Key  to 
Improved  Process  Monitoring 

Jeff  Smith  and  John  Rusnak,  Lockheed  Martin  Space 

Systems  Company 

John  Gaffney,  Perla  Unpingco  and  Joan  Weszka, 

Lockheed  Martin  Corporate  Engineering  &  Technology, 

Systems  &  Software  Resource  Center 

NDIA  CMMI®  Technology  Conference 
November  13,  2007 
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Space  Systems  C( 
Overview  ^ 

A  Space  Systems  is  engaged  in  the  design,  research, 
development,  engineering  and  production  of 
i  Satellites  a  r^dl 

i  Strategic  and  Defensive  Missile  Systems 
i  Space  Transportation  Systems 

A  2006  net  sales  of  $7.9  billion 

A  Integrated  Engineering  across  all  Space  Systems  sites 

A  Achieved  CMMI  ®  Maturity  Level  3  December  2005 
T  SCAMPIsm  VI  .1  and  CMMI-SE/SW/IPPD/SS  VI . 

T  Systems,  Software,  and  Hardware  Engineering 
T  Validated  LM-IEP  to  achieve  CMMI  compliance  ^g|| 

A  Journey  of  continual  improvement  aligned  with  Space 
Systems  business  objectives 


SCAMPI  T  Standard  CMMI  Appraisal  Method  for  Process  Improvement.  SCAMPI  is  a  service  mark  of  Carnegie  Mellon  University. 

CMMI  @T  Capability  Maturity  Model  Integration.  CMMI  is  registered  in  the  U.  S.  Patent  and  Trademark  Office  by  Carnegie  Mellon  University. 
LM-IEP  i  Lockheed  Martin  Integrated  Enterprise  Process 
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A  The  Lockheed  Martin  Space  Systems  Company  (SSC)  process 
improvement  objectives  are  aligned  with  two  major  business 
objectives: 

T  Operational  Excellence  -  quality  products/service  throughout  the  process 

T  100%  Mission  Success  -  quality  of  product  delivered 

A  Several  major  process  improvement  initiatives  are  underway 

T  Integration  of  Command  Media  across  several  major  business  sites 

T  Improving  processes  for  usability  and  scalability 

T  Integrating  tools  and  processes  and  enterprise  measurement  repositories 

A  Process  institutionalization  needs  to  be  maintained  during  deployment 
of  these  initiatives 

T  The  needs  for  measures  to  monitor  institutionalization  and  measures  of 
progress  toward  achieving  initiative  goals  were  identified 

T  Monitoring  institutionalization  measures  reduces  the  probability  of 
(process  lapsesowhile  major  improvements  are  deployed 

A  A  set  of  nBcorecardsoto  track  process  institutionalization  has  been 
developed 

T  Deployment  is  underway 
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CMMI  for  Development,  Version  1.2 

Institutionalization  T  the  ingrained  way  of  doing 
business  that  an  organization  follows  routinely  as 
part  of  its  corporate  culture 


Institutionalization  means  “the  level  of 
adoption  of  a  particular  set  of  practices  ...  is 
deep  enough,  and  broad  enough,  that  their 
use  would  continue  even  through 
organizational  and  leadership  changes.” 
Reference:  Suzanne  Garcia  and  Richard 
Turner,  CMMI  Survival  Guide,  Addison 
Wesley  2006. 


Institutionalization  T  SSC  processes 
are  used  and  are  part  of  the 
culture. 

1 .  Plan,  deploy  and  monitor  the 
processes  across  the  Enterprise 

2.  Ensure  the  proper  infrastructure 
is  in  place  to  support  these 
processes 
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Selecting  Scorecard  Measure 


A  Identified  candidate  measures 

T  Noted  measures  already  required  by  Common  Integrated 
Process  System  (CIPS)  and  used  to  gauge  institutionalization 

T  Noted  measures  related  to  CMMI  Generic  Practices 

A  Selected  measures  of  highest  importance  to  SSC 

A  Developed  a  scorecard  that  provides  an  overall  measure 
of  institutionalization  that  is  a  weighted  sum  of  values 
computed  for  each  measure 
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I  of  Scorecard  Measures 
IMI  Generic  Practices 


Name 


Process  Trained 
Personnel 


Establishing  & 
Maintaining  PDP 


Approved  PDP  & 
Program  Deployment 
Summary 


Approved 
Measurement  Plan 


Generic  Practice 


GP  2.5  T rain  People 


GP  3.1  Establish  a  Defined 
Process 


GP  2.3  Provide  Resources 
GP  2.4  Assign 
Responsibility 
GP  2.5  T rain  People 


GP  2.2  Plan  the  Process 


Selected  generic  practicesN 
focus  on  institutionalization 
of  highest  importance  to 
SSC 

1.  Training 

2.  Measurement 

3.  Compliance  . 


Name 

Generic  Practice 

Total  Measurement 
Collection  Compliance 

GP  2.8  Monitor  and 
Control  the  Process 

Measurement  Usage 

Enterprise  Standard 

Measurement  (ESM) 
Collection  Compliance 

CIPS  Process 
Compliance 

GP  2.9  Objectively 
Evaluate  Adherence 

Objective  Evidence: 
Available  vs.  Required 

GP  2.8  Monitor  and 
Control  the  Process 

CM  Mi-Based  Appraisal 
Readiness  Index 

®  CMMI  is  registered  in  the  U.S.  Patent  and  Trademark  Office  by  Carnegie  Mellon  University 
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Major  Functional  Organizations 

(MFOs) 

Program/Line  of  Business  (LOB) 

Measures  for  MFOs 

Measures  for  Programs  and  LOB 

Covers  MFOs  with  responsibility  for 
enterprise  processes  (e.g.,  systems 
engineering) 

Covers  Individual  Program 

Measures 

MFO  measures  include  level  of 

MFO  proactive  engagement  in 
process  performance  assessment 

Institutionalization  measures 
for  different  programs  in  an  LOB 
tier  up  to  that  LOB®  measures 
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Scorecard  Structure- 1 


Level 

Current  (in  Month) 
View 

Progress  (Trend) 

View 

Program 

The  component  and 
overall  score  for  a 
program  in  a  particular 
month. 

The  actual  and  planned 
overall  scores  for  a 
program  versus  month. 

LOB  (Set  of 
Programs) 

The  overall  scores  for  the 
programs  in  a  LOB  in  a 
particular  month  and  the 
averages  of  those  scores. 

The  actual  and  planned 
average  scores  for  a  LOB 
versus  month. 

Enterprise  (All  of 
the  LOBs  in  SSC) 

The  average  of  all  of  the 
LOB  average  scores, 
across  all  of  the  LOBs  in 
the  Enterprise  (SSC). 

The  actual  and  planned 
average  of  all  of  the  LOB 
average  scores  versus 
month. 

Note:  Enterprise  (All  of  the  LOBs  in  SSC)  Level  is  not  shown. 
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Scorecard  Structure-2 


Name 

Description 

Component  Data  Entry 

Process  Trained 

Personnel 

Compare  Number  of 
Program  Target 
Audience  to  Target 
Audience  Baseline 

Number  of  People  Trained 

Number  of  People  in  Target 
Audience  Baseline 

Establishing  & 
Maintaining  PDP 

Timely  Disposition  of 
the  OSP  Documents 
in  the  PDP 

Number  of  Documents 
Dispositioned  as  of  Latest 
Command  Media  Release 

Number  of  Undispositioned 
Documents  Spanning  2 
Command  Media  Releases 

Approved  PDP  & 
Program 
Deployment 
Summary 

Approved  PDP  & 
Program  Deployment 
Summary 

User  enters  "Yes"  or  "No" 

Approved 

Measurement 

Plan 

Approved 
Measurement  Plan 

User  enters  "Yes"  or  "No"  /  In  Progress 

Total 

Measurement 

Collection 

Compliance 

Compare  Measures 
Planned  to  be 
Collected  to 

Measures  Collected 

Number  of  Measures 
Collected 

Planned  Number  of  Measures 
Collected 
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PDF  Complete. 


Scorecard  Structure-3 


Name 

Description 

Component  Data  Entry 

Measurement 

Usage 

Compare  Me< 
Planned  to  be 
Measures  An< 
Reported  "Us 

asurement 
Used  to 
alyzed  and 
age" 

Number  of  Measures  Analyzed 
and  Reported  "Usage" 

Planned  Number  of  Measures 
Analyzed  and  Reported  "Usage" 

Enterprise 

Standard 

Measurement 

(ESM) 

Collection 

Compliance 

Compare  Ent< 
Standard  Me£ 
Committed  to 
the  ESM  Colie 

srprise 
asurement 
Collect  to 
acted 

Number  of  ESM  Collected 

Committed  Number  of  ESM  to 
Collect 

CIPS  Process 
Compliance 

Running  Aver; 
Findings/P  roc 
Audited 

age  of  Audit 
:esses 

Number  of  CIPS  Processes 
Compliant  (monthly) 

Number  of  Processes  Audited 
(monthly) 

Objective 

Evidence: 

Available  vs. 
Required 

Compare  Oty 
Evidence  Ava 
Objective  Evic 
Required 

ective 
ilable  to 
fence 

Objective  Evidence  Available 

Objective  Evidence  Required 
(Number  of  Processes  Performing 
that  Require  Objective  Evidence) 

CMMI-Based 

Appraisal 

Readiness 

Index 

Program  Rea< 
Completion  ol 
Appraisal  Mile 

diness  by 

FKey 

astones 

Cumulative  Number  of  Milestones 
Achieved  to  Date 

Total  Number  of  Milestones  in 
Process 
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■»  Complete 
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use  period  has  ended. 
Thank  you  for  using 
PDF  Complete. 


and  Expanded  Features 


ititutionalization  Score 


A  Map  each  component  score,  e.g.,  rPercent  of  Process 
Trained  Personnel,  Approved  Measurement  Plan.dto  its 
corresponding  utility. 

T  A  mtijityois  the  value  accorded  to  some  data  item;  the 
n/alueocan  be  numeric  (e.g.,  10)  or  linguistic  (e.g.,  yes). 

A  The  institutionalization  score  for  a  given  program  for  a 
oarticular  month  is  the  weighted  sum  of  the  utilities  for 
:he  10  component  measures. 

T  Each  rweightdrepresents  the  importance  or  value  of 
that  attribute  relative  to  those  for  the  other  9  attributes. 

A  Management  selects  the  attribute  weights  according  to 
their  view  of  the  importance  or  value  of  each  attribute. 

T  In  the  example,  the  utilities  for  each  of  the  attributes 
are  accorded  equal  weight. 
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Your  complimentary 
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PDF  Complete. 
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ilization  Score  For  Program  A 


Institutionalization  Score  for  Program  A 


Date  (Month) 


PROGRAM  SCORE  (Utility) 

82 

Score  range  (0-100);  weighted  sum  of  available 
individual  component  measures 

COMPONENT  MEASURES 

COMPONENT 

SCORE 

THRESHOLDS 

UTILITY 

1.  Process  Trained 

Personnel 

45% 

Red  -  <  30%;  Yellow  -  =  >30%  to  = 

<  50%;  Green  -  >50% 

45 

2.  Establishing  & 

Maintaining  PDP 

92% 

Red  <  90%  or  undispositioned  for 
more  than  2  Releases;  Yellow  -  90%  to 
<  100%;  Green  -  =100% 

92 

3.  Approved  PDP  &  Program 
Deployment  Summary 

Yes 

Red  =  PDP  &  Program  Deployment 
Summary  Not  Approved;  Green  =  PDP 
&  Program  Deployment  Summary 
Approved 

100 

4.  Approved  Measurement 
Plan 

Yes 

Red  =  Plan  Not  Approved,  Yellow  = 
Plan  in  Progress,  Green  =  Plan 
Approved 

100 

5.  Total  Measurement 
Collection  Compliance 

90% 

Red  <  90%,  Yellow  =>90%  to  <  95%, 
Green  =  >95% 

90 

The  component  measure  weights  are  selected  by  management.  Each  weight  is 

10%  in  this  example.  This  data  is  notional. 
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and  Expanded  Features 


ilization  Score  For  Program  A 


Institutionalization  Score  for  Program  A 


Date  (Month) 


PROGRAM  SCORE  (Utility) 


COMPONENT  MEASURES 


6.  Measurement  Usage 


7.  Enterprise  Standard 
Measurement  (ESM) 
Collection  Compliance 


8.  CIPS  Process  Compliance 


9.  %  Objective  Evidence 
Available  vs  Objective 
Evidence  Required 


10.  CMMI-Based  Appraisal 
Readiness  Index 


82 


COMPONENT 

SCORE 


90% 


Score  range  (0-100);  weighted  sum  of  available 
individual  component  measures 


92% 


90% 


80% 


THRESHOLDS 


Red  <  90%,  Yellow  =  >  90%  to 
<  95%,  Green  =  >95% 


Red  <  90%,  Yellow  =  >90%  to 
<  95%,  Green  =  >95% 


Red  -  <  80%;Yellow  -  =  >  80%  to 
=  <  90%;  Green  -  >90% 


Red  -  <  80%;  Yellow  -  =  >  80%  to 
=<  90%;  Green  -  >90% 


User  enters  a  digit  0,1, 2, 3, 4,  or  5 
corresponding  to  the  number  of 
milestones 

that  have  been  achieved. 


UTILITY 


90 


92 


90 


80 


40 


The  component  measure  weights  are  selected  by  management.  Each  weight  is 

10%  in  this  example.  This  data  is  notional. 
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and  Expanded  Features 


lalization  Score  -  LOB  Level 


LOB  Y  Institutionalization  Scores  as  of  Month  8 


Average 

1 

“  Z1 

I _ n™ 

_ i _ 

_ i _ 

_ i _ 

a> 


Program  A  Program  B  Program  C 

LOB  Y  Program 


Program  D 


This  profile  or  snapshot  shows  the  institutionalization  scores  across  a  set  of 
programs  at  one  point  in  time  as  well  as  their  average.  This  data  is  notional. 
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itutionalization  Score  T 
Program  A  by  Month 


This  graph  shows  the  institutionalization  progress  trend  fora  program  over 
a  period  of  time,  compared  to  LOB  average  results.  This  data  is  notional. 
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ization  Score  T  LOB  by  Montlr 


LOB  Y  Institutionalization  Progress 


LOB  Actuals 


Hanned 


This  graph  shows  the  institutionalization  progress  trend  for  the  programs  in  LOB  Y 
over  a  period  of  time  compared  to  planned  progress.  The  progress  measure  is  the 
average  of  the  scores  for  the  programs  LOB-wide.  This  data  is  notional. 
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*■  Complete 


Your  complimentary 
use  period  has  ended. 
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PDF  Complete. 


and  Expanded  Features 


jmmary  &  Next  Steps 


A  Summary 

T  Systematic  use  of  key  institutionalization  measures 
support  SSC  business  goals  and  objectives 

A  Next  Steps 

T  The  initial  versions  of  the  scorecards  have  been 
developed. 

T  Some  data  has  been  collected  for  several  measures. 

T  We  are  moving  toward  deployment  across  major 
functional  areas  and  lines  of  business/programs. 
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and  Expanded  Features 


Additional  Information 


A  John  J.  Rusnak:  408-742-3870  iohn  rusnak@lmco.com 


A  Jeffery  D.  Smith:  303-977-5870  rv.d.smith@lmco.com 


A  John  Gaffney:  301-721-5710 


A  Perla  Unpingco:  719-330-2033 


A  Joan  Weszka:  301-721-5714 


ioari.weszka@lmco.com 
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•  Wipro’s  Solution  -  Robust  Test  Methodology 

•  DSM/SCE/SCIM  for  Test  Strategy  &  Planning 

•  Essence  of  these  techniques 

•  Examples 

•  Orthogonal  Array  for  Test  Case  Design 

•  What  is  OA  Based  Testing 

•  Walkthrough  of  the  Wipro  OA  tool 

•  Sample  Case  studies  with  Benefits 

•  Reliability  Modeling  for  Residual  Defect  Estimation 

•  About  Reliability  Modeling 

•  Assumptions  behind  the  Wipro  Reliability  Tool 

•  Demo  of  Wipro  Reliability  Tool 

•  Summary  of  Solutions  to  Address  Testing  Challenges 


•  Q  &  A 
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Testing  Cost  Curve 


Tight  Budget 


Demanding  Customer 


Schedule  Pressure 


Testing  cost  curve:  by  William  Perry 

Optimum  test 

Cost  of  testing 


Number  of  defects 

Time 


The  Key  to  success  is  to  address  these  Challenges  as 

well  as  to  ensure  quality!! 
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and  Expanded  Features 


lenges 


A  How  to  detect  most  of  defects  early  in  the  testing  lifecycle? 
A  How  to  prioritize  on  Test  Areas? 

A  How  to  Map  resources  to  Test  Areas? 


4 

wipro 


Test  Strategy 

&  Planning 

' - J 


A  How  to  Allocate  proportionate  time  to  test  different  test  Areas? 

A  Are  all  the  paths  covered? 

(Unit  Testing  or  white  box  testing) 

A  Are  all  interfaces  tested? 

(Integration  Testing) 

A  Is  the  system  functionality  validated? 

(System  Testing) 

A  Are  all  single  mode  and  double  mode  faults  detected? 


Test  Design 
&  Execution 


A  It  is  next  to  impossible  to  do  exhaustive  testing 
(100%  possible  test  cases) 


A  How  to  ensure  that  there  is  no  redundancy  in  test  cases? 
A  When  to  Stop  Testing? 


J 


Test  Results 
Analysis 
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Lean  Management  &  Six  sigma  Techniques 


Test  Strategy  &  Planning 


ASystematic  Test  Strategy 
APrioritizing  Test  Areas 
AEarly  Defect  Detection 
AEffective  Time  &  Resource  Allocation 


Test  Design  &  Execution 


AMaximize  Test  Coverage 
AMinimize  Test  Cases 
AFoundation  for  Defect  Correlation 


Test  Results  Analysis 


AWhen  to  stop  testing 
ACertify  S/w  Quality  Levels 
AFacilitate  to  Plan  Maintenance 
Resources 


DSM  -  Dependency  Structure  Matrix  SCE  -  Software  Complexity  Estimate  OA  -  Orthogonal  Array 
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^  C  E 

m§ t0  Sfo 

S  Prioritize  the  Test  Areas  by  identifying  the  most  complex  Test  Area  to  facilitate  early 
defect  detection. 

S  Sequence  the  Order  of  Testing  of  Different  Test  Areas  so  that  there  is  minimal  delay  or 
waiting  time  between  activities  which  will  facilitate  in  Schedule  adherence. 

S  Prioritize  the  Test  Area  &  Test  Cases  in  case  of  Regression  Testing  for  optimal  use  of 
Testing  effort. 


Input/Output  Sun 

nmary  for  DSM  tool 

Objective/Tool 

Inputs 

Output 

Sequencing/DSM 

1 .  No  of  Components/Activities 

2.  Dependencies  Between  Components 
(Dependency  on  a  binary  scale) 

1 .  New  Sequence  for  activities 

2.  Cyclic  Dependency  blocks 

3.  Components/activities  that  can  be  done  in  parallel 

Test  Area 
Prioritization/SCE 

1 .  No  of  Components/Activities  &  Names 

2.  Dependencies  Between  Components 
(Dependency  on  a  5  point  scale) 

1 .  Total  Relative  Complexity  of  the  System 

2.  Individual  Relative  Complexity  of 

Modu  les/Com  ponents 

3.  Components/activities  that  can  be  done  in  parallel 

Test  Area 
Prioritization/SCIM 
(Software  Change  Impact 
Matrix  for  Regression 
Testing) 

1 .  No  of  Components/Modules  &  Names 

2.  No  of  Change  Requests/Requirements 

3.  Impact  of  each  Change  on  each  of  the 
component/Module  (On  a  5  point  scale) 

1 .  Relative  Impact  of  each  Change  on  the  whole  system 

2.  Relative  Impact  on  each  component 

3.  Total  Software  Change  Impact  Metric 
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<  yes  and  Expanded  Features 

WIPRO 


Inputs  to  DSM  for  Sequencing  Output  of  DSM  -  New  Sequence 


Component  Name 

CM 

CO 

to 

CD 

CO 

o> 

o 

CM 

CO 

in 

rH 

10 

rH 

00 

Oi 

Component  Name 

Consortium  | 

Keyword  MainteJ 

Nugget  Mainten^ 

Search  | 

Employee  NT/Xf| 

ClearTrust  | 

Client  Maintenarj 

Escalation  Matri>| 

HP  Customizatio| 

Password  Reset| 

User  Collections! 

t[AF | 

Alias  | 

POINT  SSO  | 

Announcement 

EmployeeNet 

LL  Tasks 

LL  Admin 

CP  Maintenance 

2 

6 

7 

9 

12 

19 

1 

4 

5 

8 

10 

11 

13 

14 

15 

16 

17 

18 

3 

Consortium 

2 

1 

1 

Client  Maintenance 

1 

1 

1 

1 

Keyword  Maintenance 

6 

1 

1 

1 

Consortium 

2 

H 

1 

1 

Nugget  Maintenance 

/ 

V\ 

1 

1 

1 

CP  Maintenance 

3 

n 

D 

1 

1 

1 

Search 

9 

1 

iW 

1 

Escalation  Matrix 

4 

1 

n 

L 

1 

1 

Employee  NT/XP  Login 

12 

1 

HP  Customization 

5 

n 

B 

i 

1 

1 

ClearTrust 

19 

1 

1 

Vi 

K 

Keyword  Maintenance 

6 

M 

ii 

1 

1 

1 

Client  Maintenance 

1 

1 

1 

ii 

Nugget  Maintenance 

7 

1 

1 

1 

1 

Escalation  Matrix 

4 

1 

i 

Password  Reset 

8 

■id 

1 

1 

HP  Customization 

5 

1 

1 

i 

Search 

9 

i 

i 

1 

1 

Password  Reset 

8 

1 

i 

n 

User  Collections 

10 

ll 

1 

1 

User  Collections 

10 

1 

1 

i 

TAF 

11 

h 

0 

TAF 

11 

1 

0 

Employee  NT/XP  Login 

12 

1 

Alias 

13 

1 

i 

Alias 

13 

1 

POINT  SSO 

14 

i 

i 

m 

n 

POINT  SSO 

14 

1 

1 

0 

Announcement 

15 

i 

i 

i 

Announcement 

15 

1 

1 

1 

EmployeeNet 

16 

i 

0 

EmployeeNet 

16 

1 

0 

LL  Tasks 

17 

i 

1 

LL  Tasks 

17 

1 

1 

LL  Admin 

18 

i 

■01 

LL  Admin 

18 

1 

1 

ClearTrust 

19 

i 

1 

i 

CP  Maintenance 

3 

1 

1 

1 

Output  of  DSM  -  New  Sequence  Table 


Sequence 

2 

6 

7 

9 

12 

19 

1 

4 

5 

8 

10 

11 

13 

14 

15 

16 

17 

18 

3 

levels 

i 

2 

6 

7 

9 

12 

19 

2 

1 

4 

5 

8 

10 

11 

13 

14 

15 

16 

17 

18 

3 

3 

Helps  in  identifying  components  for  concurrent 
engineering  resulting  in  optimizing  schedule 
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SYSTEM  COMPLEXITY 


SCE  Outputs 


1 

Componentl 

Componentl 

Component2 

Components 

Component4 

Components 

Components 

Component? 

Components 

Components 

ComponentIO 

Componentl  1 

Components 

Module 

contribution  to 
Complexity 

Row  ED  Col 

Row  LD  Col 

ND 

ND 

ND 

ND 

Row  LD  Col 

ND 

ND 

Row  MD  Col 

ND 

ND 

3.16 

4 

Component2 

ND 

Row  ED  Col 

ND 

ND 

ND 

ND 

Row  LD  Col 

ND 

ND 

ND 

ND 

ND 

3.65 

1 

Components 

ND 

ND 

Row  ED  Col 

ND 

ND 

ND 

ND 

Row  HD  Col 

ND 

ND 

ND 

ND 

2.51 

4 

Component4 

ND 

ND 

ND 

Row  ED  Col 

ND 

ND 

ND 

Row  HD  Col 

ND 

ND 

ND 

ND 

3.22 

3 

Components 

ND 

ND 

ND 

ND 

Row  ED  Col 

ND 

ND 

Row  HD  Col 

ND 

ND 

ND 

ND 

3.06 

2 

Components 

ND 

ND 

ND 

ND 

ND 

Row  ED  Col 

Row  HD  Col 

ND 

ND 

ND 

ND 

ND 

3.00 

2 

Component? 

Row  LD  Col 

Row  MD  Col 

ND 

ND 

ND 

ND 

Row  ED  Col 

ND 

ND 

ND 

ND 

Row  LD  Col 

3.35 

1 

Components 

ND 

ND 

Row  LD  Col 

Row  LD  Col 

Row  LD  Col 

ND 

ND 

Row  ED  Col 

Row  LD  Col 

ND 

Row  LD  Col 

ND 

4.25 

2 

Components 

ND 

ND 

ND 

ND 

ND 

ND 

ND 
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Relative  System  Complexity  &  Component  level 
Complexity  is  calculated  for  prioritization. 
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Name  of  Project 


Software  Change  Impact  Matrix 

ABC 


Contact  person 


Email 


Change  ^\ct  Matrix  AnaljsiT" 

Requests/ 

Requireme 
nts/Enhan 
cements 


Number  of  Modules 


Number  of  Test^ 


Modules/ 

Components 


Priaarj  DB 

lisUllitioi 

No 

3.82 

8.37 

CR  Impact  on  the 
sjstem 


175.09 


127.55 


76.91 


49.97 


64.99 


76.91 


49.97 


64.99 


76.91 


49.97 


813.28 


%  Coverage 


21.53% 


15.68% 


9.46% 


6.14% 


7.99% 


9.46% 


6.14% 


7.99% 


9.46% 


6.14% 


A  Quantitative  Framework  for  Test  Area 
Prioritization  &  Effort  Allocation. 
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OA  Based  Testing  is  a  methodology  which  facilitates  in  ensuring  a  higher  coverage  of 
Testing  the  Possible  causes  of  failure  with  a  lower  number  of  Test  Cases 


\ 

The  first  Step  in  OA  based  Test  design  is  to  parameterize  the  Test  Area 
into  Factors  &  levels 

_ J 


\ 

Once  factors  &  levels  identified  are  fed  into  Wipro’s  OA  tool,  Test  Runs  are 

automatically  generated  which  reduces  the  test  case  writing  time. 

\ _ J 


f  \ 

OA  ensures  that  all  levels  of  each  factors  are  tested  at  least  once  &  all  possible  pair 

wise  combinations  of  factors  are  tested  at  least  once. 

\ _ ) 


f  \ 

Wipro’s  OA  application  experience  indicate  benefits  of  significant  reduction  in  total 

testing  effort  or  significant  improvement  in  test  coverage  of  possible  failure  modes. 

\ _ _ _ J 
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AOnce  Factors  &  Levels  are 
identified,  test  sets  are  generated 
by  the  tool 

AAII  levels  of  each  factor  are 
tested  at  least  once.  (All  single 
mode  failures  re  covered) 

Mil  possible  pairwise 
combinations  are  tested  at  least 
once  (All  Double  mode  failures  are 
covered  at  least  once) 
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□  Project  Objective: 

Y  To  optimize  test  cases 

□  Project  Goal  : 

T  To  Improve  test  coverage, 
defect  detection 

□  Methodology  Used  :  DSSS+ 

T  Orthogonal  Array 

A  Results: 

T  Full  factorial  test  cases  : 
58632 

T  OA  based  test  cases  :535 
T  After  adding  a  few  more  test 
cases  :  563 


A  Business  Benefits 

Y  Test  case  reduction  by  35% 

Y  New  bugs  found  using  OA  T  6 
T  All  Defects  found  in  previous 

releases  detected 


OA  Applied  in  retail  domain 


m 


□  Project  Objective: 

T  Optimize  the  number  of  test 
cases 

□  Project  Goal  : 

□  Reduce  testing  effort 

□  100%  functionality  coverage 

□  Methodology  Used  :  DSSS+ 

□  Orthogonal  Array 

A  Results 

T  848  test  cases  reduced  to  167 


A  Business  Benefits 

i  Able  to  complete  2  complete 
cycles  of  testing  in  planned 
time 

i  Test  case  reduction  by  80% 

i  100%  functionality  covered 

T  Validated  with  requirements 
trace  ability  matrix 


Wipro  confidenti; 


OA  Applied  in  Telecom  -  Project  2 


WIPRO 


□  Project  Objective: 

T  To  optimize  test  cases 

□  Project  Goal  : 

T  To  Improve  test  coverage, 
defect  detection 

□  Methodology  Used  :  DSSS+ 

T  Orthogonal  Array 


A  Business  Benefits 

T  Earlier  release  had  17  test 
cases  and  did  not  detect  any 
bugs 

T  For  the  same  release  with  OA 
generated  24  test  cases,  the 
test  coverage  and  defect 
detection  improved  drastically 

T  Test  Coverage  improved  from 
A  25  to  51  %  in  service 
option. 

A  50  to  70%  in  protocol 
revision 

A  .40  to  100  %  in  service 
negotiation.  2 
A  5  to  40  %  in  service 
handoff 


OA  Applied  in  Finance  domain 


WIPRO 


□  Project  Objective: 

T  Optimize  the  number  of  test  cases 


A  Business  Benefits 

T  Reduced  testing  effort  by  56% 


□  Project  Goal  : 

T  Reduce  testing  effort 

□  Methodology  Used  :  DSSS+ 

T  Orthogonal  Array 

□  Results 

T  Full  factorial  :  548  test  cases 
T  rdExpert  ( industry  product) : 
290tc 

Y  Wipro  OA  tool  :  236tc 


A  Concluding  remarks  of  the 
practioner 

T  Wipro®  OA  tool  generated  lower 
test  cases  when  the  factors  and 
levels  are  high 

T  Despite  the  lower  number  of  test 
cases  the  coverage  is  more  when 
compared  with  rdExpert  ( industry 
standard  tool  ) 
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S/W  Reliability  Modeling 


Static  Modeling 


Dependent  Project  Attributes 


Dynamic  Modeling 


Y  =  f  (xl,  x2,  x3,  x4)  +e 

Total  Dev  Phase 

Testing  Phase 

Rayleigh  Model 


Exponential  & 
other  models 


NHPP& 
Other  Models 


Dependent 

Eg:-  Defect  rate 

Independent 

Complexity,  skill  level,  etc 

Based  on  Past  Project  Data 

Based  on  Current  Project  Data 
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□  Effort  in  Testing  is  homogeneous  throughout  the  testing  phase. 

□  Since  this  assumption  is  not  always  applicable,  normalization  of 
defect  data  wrt  test  cases  is  required. 

□  An  acceptable  amount  of  coverage  is  achieved  by  the  test  cases  under  use 
(  Use  of  OA  based  test  case  design  or  other  robust  methods  is  assumed) 

□  The  time  sequence  of  the  defect  data  should  be  maintained 

□  At  least  75%  of  the  testing(  test  case  execution)  should  be  complete 
for  predictive  validity  AND 

□  A  plot  of  the  defect  rate  should  indicate  a  declining  defect  trend 


Tool  is  not  to  be  used  for  defect  estimation  without 

Test  Execution. 
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Model 

Delayed  S 

MSE 

Alpha 

Beta 

Confidence  Interval 

Accuracy 

Tested 

Untested 

Total  Estimated  Defects 

Total  Defects  Detected 

Total  TCs  Executed 

Residual  Defects 

Additional  TCs  Required 

TC 

Defects 

% 

\ 
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\ 

\ 

y~  ~1 

V  1 

\= — 1 
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11 


12 


13 


14 


15 


16 
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18 


19 


20 


26 


Day 


TC 


Cum  Failures 


10 


24 


49 


79 


134 


250 


500 


12 


900 


22 


1012 


45 


Estimate  Remaining 
Defects 

Predict  Defec 
Specified  Test 

Estimate  Defei 
Untested  Fea' 


1115 


55 


1119 


58 


Option  1  -  Estimate  Remaining  defects 
On  click  of  this  button,  a  dialog  box  pos  up  which  asks  for  the  required 
confidence  level.  Once  that  is  specified,  the  tool  gives  the  output  as 

shown  in  next  slide 
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55 
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1st  table  gi 

H 

1 

J 

K 

L 

M  - 

Model 

Delayed  S 

MSE 

7.432118145 

Alpha 

82 

Beta 

0.2 

Confidence  Interval 

14.5667 

Accuracy 

0.8503 

Tested 

Untested 

Total  Estimated  Defects 

82 

Total  Defects  Detected 

58 

Total  TCs  Executed 

1119 

Residual  Defects 

24 

Additional  TCs  Required 

3245.1 

TC/  \ 

Defects 

% 

/  \  1763 

67 

81.71 

V  \  2417 

76 

92.68 

\  3066 

79 

96.34 

\  3715 

81 

98.78 

\  4364 

82 

100.00 
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Estimate  Remaining 
Defects 


Predict  Defects  for 
Specified  Test  Cases 


Estimate  Defects  for 
Untested  Features 


<4  ► 


for  estimation. 

2nd  table  gives  the  details  of  the  remaining  defects  &no  of  test  cases  to 
be  executed  to  find  out  the  remaining  defects. 

3rd  table  gives  the  estimated  breakup  of  testcases  to  be  executed  & 
remaining  defects  that  will  be  found  out 

►i  \  Reliability  Plot  \Data  / 
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Summary 

of 

Solutions  to  Address 
Testing  Challenges 
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OATool 

Orthogonal  Array 

Expanded  Features 

Test  Phase  :  Test  Optimization 


Features 

A  Systematic  and  statistical  method  of  pair¬ 
wise  combinations  of  selected  factors  or 
variables  across  their  levels. 

A  Creates  an  optimized  test  suite  with  lesser 
test  cases. 

A  Detects  all  single  mode  and  double  mode 
defects. 

A  Increases  confidence  level  in  the  system 
by  executing  a  concise  set  of  tests  and 
uncovering  most  of  the  bugs. 


Benefits 

A  Helps  in  productivity  improvement  with  cycle 
time  reduction. 

A  Helps  in  improving  the  test  coverage. 

A  Helps  in  minimizing  the  size  of  test  suite  by 
eliminating  the  redundant  test  cases  from 
the  test  suite. 

A  Test  effort  reduction  in  terms  of  test  case 
writing  and  execution. 


Client  Name 


Case  Study 

A  large  North  American  telecom  equipment  manufacturer 

Project  Scope: 

A  Testing  of  a  large  IP-PBX  system. 

A  Live  Communication  Version 
features  to  be  incorporated. 

A  Initial  test  suite  contains  more 
than  800  test  cases. 


Challenges  : 

Optimizing  the  test  suite 
without  compromising  on 
the  test  coverage. 


Benefits  : 

Considerable  amount  of  saving  in 
terms  of  test  effort  and  time. 

-  The  number  of  test  cases 
was  reduced  from  800  to  170  . 

-  There  was  a  reduction  in  approx. 
75%  of  the  testing  effort. 

-  No  compromise  on  test 
coverage 
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Comolexitv  Deoendencv  Chanae  imoact 

an  integrated  tool  consists  of  DSM, 


DSM  Tool  (Dependency  Structure  Matrix) 

Test  Sequencing 

Dependency  Structure  Matrix  analyses  the 
dependencies  among  the  modules  and 
Helps  project  managers  in 

-  Determining  the  sequence  of 
test  execution  of  the  modules. 

-  Deciding  which  modules  should  be 
kept  under  a  single  team. 

-  Deciding  which  modules  can  be 
executed  in  parallel  without  any 
dependency  clash. 


Which  modules  should  be  tested 
first? 


Tool  (System  Complexity  Estimator) 

Effort  Estimation 

System  Complexity  Estimator  analyses  the 
complexity  and  the  dependency  of  modules 
in  a  system  and  helps  project  managers  in 
estimating  testing  effort  distribution  across 
the  modules. 

Which  module  requires  maximum 
testing  effort? 


SCIM  Tool  (System  Change  Impact  Matrix) 

Maintenance  Phase 

System  Change  Impact  Matrix  analyses  the 
system  complexity,  and  the  impact  of  each 
Change  Request  (CR)  on  all  the  modules  in 
a  system  and  helps  project  managers  in 

-  Estimating  the  relative  test  effort 
distribution  across  modules  during 

maintenance  phase. 

-  Estimating  the  relative  test  effort 
distribution  across  different  CRs. 


Which  module  requires  maximum 
attention  from  a  change  perspective? 


Client:  A  large  North  American  server  and  storage  manufacturer. 

-  Before  using  DSM  around  871 

Project:  Asset  Management  Systems  (Maintenance  Project) 

regression  test  cases  were 

Project  Scope 

-  to  avoid  unnecessary  repetition  of 

executed.  However  during  the 

-  to  reduce  the  test  execution 

r'\/rklo  timo  of  tho  roloaco 

test  cases. 

-  to  ensure  that  there  is  no  defect 
slippage  because  of  the  reduced  set 
of  regression  test  cases. 

beta  testing  conducted  in  Feb 

2007,  only  208  optimized  (DSM) 
regression  test  cases  were 
executed  bv  avoiding  duplication. 

OyUlfci  unit;  Ul  lilt;  ItJItiaot;. 

Challenges 

-  to  find  the  correct  sequence  of 
execution  of  modules. 

Benefits 

-  Helped  in  determining  the  sequence  of 
execution  without  anv  deoendencv  clash 

-  The  test  cycle  time  got  reduced  from 
20  person  davs  to  12  person  davs. 
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Test  Reporting 


This  tool  helps  project  managers  in 

-  systematically  analyzing  various  metrics  applicable  in  a 
testing  project  faster  and  thus  with  less  effort. 

-  standardizing  reports  generated  across  projects  by 
providing  graphical  and  tabular  representation  of 

^Defect  T  rend 

A"est  case  productivity,  Pass  rate,  test  efficiency 
^Defect  priority  analysis  etc. 


Reliability  Analysis 


This  tool  analyses  the  trends  of  Defect  Detection  in  a  test 
cycle  and  helps  the  test  manager  in 

-  Estimating  residual  defects  in  the  system. 

-  Deciding  when  to  stop  Testing  of  a  system. 


Case  Study 

Client:  A  large  North  American  Telecom  Equipment  manufacturer. 


Project  Scope 

-  verification  of  leading  north 
American  equipment  vendor® 
element  management  system  for  his 
broad  band  access  products. 

Challenges 

-  whether  to  release  product 
or  continue  testing. 


-  to  release  products  in  the 
field  with  a  level  of  confidence. 

Benefits 

-  Predicted  number  of  defects 
in  the  past  were  validated  by 
the  response  from  the  field. 


-  Helped  in  taking  a 
decision  on  whether  to 
continue  testing  or  release 
the  feature. 

-  Based  on  the  reliability 
output,  recommendation 
was  given  to  stop  the 
general  availability  of  one 
of  the  release. 
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■  Environment  of  an  Acquisition  Office 
SCAMPI 

■  The  Program 

■  Programmatics 

■  The  Organization 

■  The  Timeline 

SCAMPI-C  Outcomes 
Utilization  of  SCAMPI-C  Outbrief 
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Program 


■  Space-Based  Radar  (SBR)  is  a  DoD  program  for 
a  constellation  of  radar  satellites  organized  as  an 
Air  Force  program  under  the  Space  and  Missile 
Systems  Center  (SMC)  as  the  lead  execution 
agency  (PEO)  and  USECAF(Space)  as  the 
milestone  decision  authority 

■  Critical  partner  agencies  were  on 
management  boards,  but  lacked  direct  ties 
into  PMO  staff  at  SMC 

SBR  was  projected  to  be  a  US$20B  program  with 
10+  years  until  first  launch 
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Programmatics 


■  SBR  was  in  Phase  A  (NSS-0301  life  cycle)  Concept 
Development  with  two  competing  prime  contractors 

■  USAF  management  by  Colonel  SPD  and  3-Star  PEO 

■  SBR  Engineering  Challenges:  applying  systems 
engineering  principals  across  JPO  and  independent 
partner  agencies,  high  risk  technologies  in  baseline 

■  SBR  had  created  a  family  of  plans,  two  of  which  called 
for  acquisition  process  improvement:  the  Software 
Acquisition  Management  Plan  (SAMP)  and  the  System 
Engineering  Management  Plan  (SEMP  -  in  draft) 

■  SCAMPI-C's  based  on  the  CMMI-AM  were  to  be  used  as 

a  measurement  tool  to  guide  process  improvement 
efforts  , 
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Undersecretary  of  the  Air  Force  for  Space  (USECAF(Space)) 
Milestone  Decision  Authority  (MDA) 
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PI  Timeline 


■  August  2004  SAMP  directed  SCAMPI 


■  October  2004  SPD  charter  to  SCAMPI  team 


■  December  2004  SCAMPI  team  formed  and 
document  inventory  begun 

■  21  Feb  2005  initial  SCAMPI  brief/start  SCAMPI 

■  1  March  2005  New  Management  Structure 
Announced 


4  March  2005  Complete  SCAMPI  interviews 
10  March  2005  SCAMPI  outbrief 
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■  In  March  2005,  SBR  was  formally  changed  to 
become  Space  Radar  (SR)  under  (then)  BGen 
Sheridan  as  his  own  PEO  and  SPD  under 
USECAF(Space)  as  MDA 

■  Program  office  was  to  be  slimmed  down  and 
moved  from  SMC  to  the  Washington  DC  area  to 
be  closer  to  partner  agency  representatives 

■  Agencies  would  provide  staff  members  into 
the  new  "Integrated  Program  Office" 
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■  SR  goals 

■  Consolidate  management  and  budget  execution 

■  Program  was  made  the  responsibility  of  a  new  PEO 

■  PEO/SPD  combined  under  single  USAF  general  officer 

■  New,  leaner  technical  baseline 

■  Geographically  move  program  office  to  be  closer  to 
stakeholders 


■  SR  challenges 

■  "Consolidate",  "Geographically  Move",  "new  ...  baseline" - 
Continue  operations  without  interruption  during  these 
transitions 


■  SR  engineering  challenges 


Rescope  contracts,  re-validate  technical  baselines,  maintain 
and  improve  engineering  capabilities 
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PI-C  Outcomes 


The  SCAMPI-C  of  the  SBR  program  identified  goals  for 
process  improvement  as  well  as  process  assets  that  should 
be  nurtured 

General  findings  were  that  most  process  areas  were 
operating  at  level  2 

■  Contracting  was  found  to  be  executing  at  or  above  level  3 

Broadly,  several  process  assets  were  identified  in  the 
process  creation 

■  Several  staff  members  were  noted  for  their  ability  to  create 
consensus  processes  of  high  capability  and  maturity 

Subject  Matter  Experts  were  less  communicative  about 
their  jobs  after  the  announced  management  change,  due 
to  uncertainty  if  they  would  be  involved  in  the  new 
program  or  be  reassigned 
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Moving  the  PMO  resulted  in  a  need  to  decide  who  would 


move 


■  SCAMPI  identified  which  processes  were  effective,  so 
the  staff  who  developed  those  processes  were 
identified  as  key  program  personnel 

■  Personnel  who  wouldn't  be  able  to  be  moved 
triggered  a  program  risk  to  be  created  and  then 
mitigated  via  a  staffing  plan 

■  When  personnel  slated  to  move  were  subject  to  a 
staffing  reclama  by  SMC,  SR  was  able  to  demonstrate 
the  direct  impact  of  those  staff  members  being 
unavailable 
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■  Several  open  issues  were  identified 

■  Several  specific  practices  needed  their 
processes  to  be  defined  (in  the  CMMI  sense) 
via  the  creation  of  management  plans  and 
operating  instructions  to  achieve  level  3; 
prioritization  could  be  given  to  which 
instructions  based  on  the  observations  of  the 
SCAMPI  team 

■  The  maturity  of  the  various  process  areas  gave  a 
set  of  metrics  to  management's  transition  team 
to  base  decisions  on  staff  composition,  size,  and 
location 
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■  SCAMPI-C  based  on  the  CMMI-AM  was  useful 


■  Despite  a  significant  management  change 

■  Results  of  the  SCAMPI-C  were  usable  by  the  new 
management 

■  Despite  some  changes  to  the  technical  baseline  of  the  program 

■  Mechanism  remains  a  very  useful  tool  for  maintaining 
insight  into  the  acquisition  organization  during  changes  in 
its  lifecycle 


■  Overall  observation:  anticipated  or  actual  changes  in  a 
program  are  no  reason  to  avoid  a  SCAMPI;  indeed  a 
SCAMPI  may  even  inform  decision  makers  during  such 
periods  of  uncertainty 
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Software  Engineer 
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Los  Angeles,  CA  900002- 
2957 

Francis.J.Sisti@aero.org 

www.aero.org 

(0)  1-310-336-1502 

(C)  1-602-538-6554 

(Fax)  1-310-726-1332 


DeWitt  Latimer, 

Captain,  U.S.  Air  Force 

Software  Engineering,  PhD 
Student  (AFIT/CIGD) 

use 

941  W.  37th  Place 
SAL  300 

Los  Angeles,  CA  90089-0781 

dlatimer@USC.edu 
http :  //robotics,  use.  ed  u 
(C)  1-310-722-8157 


I  .»PDF 

t  Complete 


Your  complimentary 
use  period  has  ended. 
Thank  you  for  using 
PDF  Complete. 


7**  PDF 

*  Complete 


use  period  has  ended. 
Thank  you  for  using 


Your  complimentary 


PDF  Complete. 


jnyms 


■  CMMI-AM  -  CMMI  -  Acquisition  Module 

■  PEO  -  Program  Executive  Officer 

■  PMO  -  Program  Management  Office 

■  MDA  -  Milestone  Decision  Authority 

■  SBR  -  Space-Based  Radar  (Program) 

■  SMC  -  Space  and  Missile  Systems  Center 

■  SPD  -  System  Program  Director 

■  SR  -  Space  Radar  (Program) 

USECAF(Space)  -  Undersecretary  of  the  Air 
Force  for  Space 
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■  CMMI-Acquisition  Module  (CMMI-AM),  Version 
1.0,  dated:  February  2004 

■  NSS  0301,  National  Security  Space  Policy 
Directive  03-01,  dated:  27  December  2005 

■  Standard  CMMI(SM)  Appraisal  Method  for 
Process  Improvement  (SCAMPI(SM)),  Version 
1.1:  Method  Definition  Document,  Dated: 
December  2001 
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PM  seeks  to  ascertain  fidelity  to  CMMI  among  bidders;  would 
like  to  encourage  most  capable  to  bid  as  rprimesoand 
encourage  others  to  partner. 

PM  has  had  good  experience  using  CMMI  and  SCAMPI-B  for 
previous  source  selection 

PM  needs  to  minimize  time  and  effort  to  gather  the  data 

SEI  called  in  by  SETA  supporting  PMO  to  develop  a  strategy 
that  will  work  under  these  contraints 
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RFP  to  be  released  by  NSA  Program  for  software-centric 
infrastructure  management  system. 

16  vendors  express  interest  in  competition  through 
participation  in  pre-RFP  workshops 

Program  is  looking  for  rLevel  3ofor  potential  winner 

All  potential  bidders  have  rraised  their  hando 
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Publish  a  Request  For  Information  seeking,  among  other  things, 
objective  evidence  oriented  to  CM  Ml  process  areas 

Use  the  FAR  Multi-step  Process  (15.202)  to  conduct  an  evaluation 
and  advise  certain  offerors  that  they  are  highly  competitive.  Note:  all 
others  may  still  bid  with  no  prejudice. 
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Help  the  program  narrow  the  scope  of  CMMI  process  areas  to  those 
most  critical  for  success 

Quickly  assemble  an  evaluation  team  and  train  them  in  CMMI 
(refresher)  and  SCAMPI 

Produce  an  evaluation  plan  and  communicate  to  offerors 
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Firste 

A  rPIID  Matrixolisting  appropriate  artifacts  for  each  CMMI 
practice  within  scope 

Thene 

The  actual  artifacts  for  review  by  the  SCAMPI  Team 
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Category 


Process 

Management 


Project 

Management 


Support 


Process  Areas 


Organizational  Process  Focus 
Organizational  Process  Definition 
Organizational  Training 
Organizational  Process  Performance 
Organizational  Innovation  and  Deployment 


Project  Planning 
Project  Monitoring  and  Control 
Supplier  Agreement  Management 
Integrated  Project  Management  for  IPPD 
Risk  Management 
Integrated  Teaming 
Integrated  Supplier  Management 
Quantitative  Project  Management 


Configuration  Management 

Process  and  Product  Quality  Assurance 

Measurement  and  Analysis 

Decision  Analysis  and  Resolution 

Organizational  Environment  for  Integration  Causal 

Analysis  and  Resolution 
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Scope  of  the  Evaluation 


Eight  of  the  original  sixteen  organizations  chose  to  participate  in  the 
evaluation 

This  was  not  known  until  the  first  day  of  the  evaluation 

Three  projects  from  each  organizations  were  to  be  evaluated 

Needed  to  scope  the  activities  to  match  the  available  resources 

Used  variation  of  Nominal  Group  Technique  to  assist  team  in  selection  of 
critical  Specific  and  Generic  Practices  from  CMMI  VI  .1 
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Requirements  Development:  10  Specific  Practices 
Technical  Solution:  9  Specific  Practices 
Product  Integration:  9  Specific  Practices 
Generic  Practices  (CL3):  36  Practices 
Multiply,  by  8  Business  Units  and_  3  projects  per  BU 


1536  practices  to  be  characterized  in  approximately  20  hours! 
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Three  Dimensions 


CMMIV1.1  Process  Areas 

A  Most  critical  to  program  success 

A  Specific  (performance)  and 

generic  (institutionalization)  practices 

Target  Capability  Level 

A  Indicated  by  fidelity  to  Generic  Practices  in  each  PA 
A  Target  is  Capability  Level  3 
6  Organizational  processes 
6  Tailored  for  program  use 
6  Stakeholder  involvement 
6  Monitoring  and  control 
6  Driven  by  policy 
Past  Appraisal  Data 

A  Appraisal  Disclosure  Statement 
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Specific  Practices 


Requirements  Development 

A  Goal  1,  Develop  Customer  Requirements, 
is  excluded 

A  Establish  Product  Requirements 
A  Allocate  Requirements  to  Components 
A  Identify  Interface  Requirements 
A  Establish  CONOPS  and  Scenarios 
A  Validate  Requirements 
Technical  Solution 

A  Evolve  CONOPS  and  Scenarios 
A  Select  Product  Solutions 
A  Design  the  Product 
A  Establish  Tech  Data  Package 
A  Design  Interfaces 
A  Implement  the  Product  Design 


Product  Integration 

A  Establish  Integration  Environment 
A  Manage  Interfaces 

A  Confirm  Product  Readiness  for  Integration 
A  Assemble  the  Product 
A  Evaluate  the  Assembled  Product 
A  Package  and  Deliver  the  Product 

Generic  Practices  for  Each  PA 

A  CL3:  Establish  and  Maintain  Organizational 
Processes 

A  CL2:  Plan  the  Process 
A  CL2:  Involve  Relevant  Stakeholders 
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Requirements 

Development 

Requirements  Specifications 

Allocation  Tables 

Requirements  Traceability  Matrices 

Interface  Control  Documents 

SRR  Presentations 

Technical 

Solution 

CONOPS 

Use  Cases  and  Scenarios 

Unit  Development  Folders 

Source  Code 

Rack  Elevations 

Product 

Integration 

Integration  and  Test  Plans 

Integration  Test  Results 

Pre-Ship  Checklists 

System  Inventory 

Shipping  Documentation 

Embedded  SCAMPI-C  at  NSA 

Software  Engineering  Institute  CamegieMellon  Joe  Wickless,  November,  2007  13 

©  2007  Carnegie  Mellon  University 


-vpnF 

■»  Complete 


Your  complimentary 
use  period  has  ended. 
Thank  you  for  using 
PDF  Complete. 


and  Expanded  Features 


imjvujw  ui  Objective  Evidence 


Each  offeror  supplies  the  actual  artifacts  for  the  CMMI®  practice  listed 
in  the  Pll  matrix  submitted  earlier. 

Appraisal  team  makes  a  judgment  based  on  the  artifact  and 

rbharacterizesothe  practice  for  each  project  by  CMMI®  process 
area. 

Appraisal  team  aggregates  the  characterization  to  the  BU  level . 

This  is  the  same  process  used  during  a  SCAMPI-ASM  with  much  less 
evidence,  coverage  and  rigor. 
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The  intent  of  the  model  practice  is  judged  to  be  absent  or  poorly 
addressed  in  the  approach  or  deployment.  Goal  achievement  is 
judged  unlikely  because  of  this  absence  or  inadequacy. 


The  intent  of  the  model  practice  is  judged  to  be  partially 
addressed  in  the  approach  or  deployment.  Only  limited  support 
for  goal  achievement  is  evident. 

The  intent  of  the  model  practice  is  judged  to  be  adequately 
addressed  in  the  set  of  planned  or  deployed  practices,  in  a 
manner  that  clearly  supports  achievement  of  the  goal  in  the 
given  process  context. 
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_ HI  Practice  Characterization 


for  the  Business  Unit  (BU) 


The  BU  Characterization  is  Red 

Awhen  at  least  one  of  the  instances  is  characterized  Red 

The  BU  Characterization  is  Yellow 

-  when  none  of  the  instances  are  characterized  Red,  and  fewer  than  two  of  the 
instances  are  characterized  Green 

The  BU  Characterization  is  Green 

-  when  at  least  two  of  the  instances  are  characterized  Green  and  none  of  the  instances 
are  characterized  as  Red. 
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Business  Unit 


Project  A 
Project  B 
Project  C 
OU  Aggregate 


Requirements  Development 


Technical  Solution 


Product  Integration 


Business  Unit 

Project  A 
Project  B 
Project  C 
OU  Aggregate 

Business  Unit 


Project  A 
Project  B 
Project  C 
OU  Aggregate 


Requirements  Development 


Technical  Solution 


Product  Integration 


Requirements  Development 


Technical  Solution 


Product  Integration 
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rasi  Appraisal  Data 


Offerors  were  asked  to  submit  relevant  Appraisal  Disclosure 
Statements  (ADS)  dated  within  the  last  3  years 

An  ADS  is  generated  each  time  that  a  SCAMPISM-A  appraisal  is 
conducted.  The  only  way  for  a  business  unit  to  obtain  a  CMMI® 
Capability  or  Maturity  Level  is  to  conduct  a  SCAMPISM-A  . 

Every  SCAMPISM-A  consists  of  a  model  scope  (CMMI®  process 
areas)  and  an  organizational  scope  (sample  projects  and 
support  groups) 

The  current  ADS  in  use  leaves  much  room  for  variability  in  the 
amount  of  insight  provided,  thus  the  need  for  some  expert 
analysis 
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Past  Appraisal  Data:  Example 


Submitted  ADS  (1 1/04)  for  the  business  unit  ABC  Systems. 

Achieved  ML3  which  represents  equivalence  of  CL3  in  PA  scope. 

No  projects  listed  on  ADS.  Cover  letter  indicates  that  one  submitted 
project  rparticipatedo 

Lead  by  consultant  lead  appraiser  with  additional  lead  appraiser  on  the 
team. 

Very  little  contextual  information  in  ADS. 
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Past  Performance 


Aggregate 

(Using  Process  as  a  gate) 


Weight 

Evaluation  Criteria 

Three 

Four 

Five 

Six 

Seven 

0.2 

1 .  Recent  performance  record  in  developing, 
producing,  and  delivering  rapid  prototypes  or 
capabilities  in  Spins  or  Spirals. 

10 

2 

02 

0.2 

10 

2 

5 

1 

0.2 

2.  Recent  performance  record  in  transitioning 

software  prototypes  to  operational  status. 

10 

2 

5 

1 

02 

0.2 

5 

1 

0.2 

3.  Recent  performance  record  in  developing  a 
service  oriented  architecture  with  a  foundation  of 
logical  core  services  that  enables  the  system  to  be 
extended  over  time. 

5 

1 

0.2 

10 

2 

10 

2 

10 

2 

0.2 

4.  Recent  performance  record  in  the 
development  of  a  scalable,  extensible,  and  modular 
design  that  accommodates  new  interfacing  systems, 
changes  to  interfaces,  and  growth  opportunities  to 
enhanced  protection  levels. 

10 

2 

0.2 

5 

1 

5 

1 

10 

2 

0.2 

6.  Recent  performance  record  in  integrating  new 
COTS  capabilities  into  customer  legacy  systems 
with  EITC  and  NSA  Enterprise  Architecture 
compliance. 

02 

0.2 

02 

0.2 

5 

1 

1 

Total  Scores 

7.2 

3.6 

5.4 

7 

Rating 
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Competitive 

Highly  Highly  Highly 

Competitive  Competitive  Competitive 

GREEN  =  10 
YELLOW=5 


Embedded  SCAMPI-C  at  NSA 

Software  Engineering  Institute  CamegieMellon  Joe  Wickless,  November,  2007  22 

©  2007  Carnegie  Mellon  University 


>PDF 

t  Complete 


use  period  has  ended. 
Thank  you  for  using 


Your  complimentary 


PDF  Complete. 


r rade  to 

and  Expanded  Features 


Lessons  Learned 


The  Advisory  Multi-step  Request  For  Information  (RFI)  is  an 
excellent  mechanism  for  identifying  contractor  viability. 

Using  the  CMMI©  and  SCAMPISM  to  verify  process  maturity  of 
future  offerors  during  the  RFI  evaluation  was  valuable  and  proved 
to  be  a  useful  rtjatingdmechanism 

If  Appraisal  Disclosure  Statements  rate  very  high  in  relation  to 
timeliness,  correlation  of  sample  projects,  CMMI  scope,  etc,  use 
that  project  data  versus  inspecting  CMMI  artifacts 

Offerors  should  be  advised  that  they  have  the  option  to  also 
submit  final  findings  or  other  contextual  information  to  accompany 
any  relevant  ADS 

Offerors  should  be  given  a  firm  requirement  for  data  format  (e.g. 
CD,  DVD,  etc)  and  not  allowed  to  deviate 
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Contact  Information 


Joe  Wickless 

Software  Engineering  Institute 
4301  Wilson  Blvd 
Arlington,  VA  22203 
443-690-2026 
joew@sei.cmu.edu 
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Lessons  Learned  frame 


A  5  organizations:  Jacobs  and  Government 
A  About  75  projects 
A  Over  three  years 
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What  we  will  discusse 


A  Perspectives  of  project  performance 

A  Relationships  of  performance  to  process 
capability 

T  Project  level 
T  Organizational  level 

A  How  Lean  can  help 
A  Conclusions  and  summary 
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Chief  Engineer,  Jacobs  Technology/ITSS 
SCAMPI  Lead  Appraiser 
(lean)  Six  Sigma  Black  Belt 

Member  of  the  Steering  Committee  and  Co-Chair  of  the  Software 
Committee,  NDIA  Systems  Engineering  Division 

Visiting  Scientist,  Software  Engineering  Institute 

Experience: 

T  Project  Manager,  Software  Development 

T  Systems  Engineer,  Advanced  Research  Center 

T  Program  Manager  T  Air  Force  Systems  Acquisition  and 
Development 

T  B.S.  Aerospace  Engineering 
T  M.S.  Operations  Research 
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What  may  be  causing  the  variability? 
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Variability  ine  . 

Project  complexity? 


Why  not  just  focus  on  improving  project  performance? 
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Project  Profitability  (constant  dollars) 
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How  mature/capable  is  the  process? 
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How  may  the  process  be  performing? 


This  graph  implies  a  positive  correlation  between 
process  capability  and  project  performance 

Can  we  tell  if  process  capability  is  CAUSING 
an  effect  on  project  performance? 


Institutionalization? 
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Degrees  of  Coupling.... 

between  process  capability  and  project  performance) 


A  Completely  unknown  and  uninvestigated  relationship 
between  process  capability  and  project  performance 

A  Multi-variate  correlation  of  process  capability  and  project 
performance 

T  Heuristic  at  ML  2  and  3 

A  (perhaps  gross  correlations  to  CL  or  ML) 

i  Statistical  at  ML  4  and  5 

A  (performance/quality  attribute  to  sub 
process  capability) 

A  Causal  analysis  of  project  performance  based  in  part  on 
process  capability 

A  Process  capability  DRIVEN  by  project  performance 
goals 

T  and  enabled  by  ORGANIZATIONAL  process-performance 
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ML  3 
ML  4 
ML  5 
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Relationships  Between  Process  &  Performance 


A 

A 

A 


Projects  can  perform  well  without  mature  or  capable 
processes 

T  Just  not  consistently 

Capable  processes  are  NO  guarantee  of  project 
performance 
i'  (even  at  ML  4  and  5) 

To  guarantee  that  our  process  improvement  effort  will 
result  in  improvements  in  project  performancee 

V  We  are  led  to  what  conclusion? 
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How  can  Lean  help? 


Link  project  performance  to  process  capability  early? 


A  Lean  (Kaizen)  rapid  improvement  events 

i  Are  based  on  initial  and  target  performance  measures 
i  Are  driven  by  the  customer  view 

A  (i.e.  reflect  business  needs  from  the  project®  perspective) 
i  Include  project  workflow  performance  measures 
A  Current  state 
A  Goal  state 
A  Ideal  state 

A  Examples  of  Lean  project  performance  measures: 

Product  Cycle  time  Process  Performance 

Defect  profiles  (life  cycle  phase/time  in  system)  Product  Performance 
Degree  of  synchronization  Process  Performance 

Waste  eliminated  (dollars/life  cycle)  Process  Performance 


Map  to  process  capability? 
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A  Alternative  practices 

T  rLeanedospecific  or  generic  practices  are  supported 
T  May  end  up  with  a  greater  number  of  SPs 

A(e.g.  Project  Planning) 

A  Process-performance  (High  Maturity) 

T  Process-performance  baselines 
T  Process-performance  models 
T  Could  well  include  lean  process-performance 

A  Non-model  findings 

'i  Lean  attributes 

A  Consistency  of  iterations 
A  Waste  identified 
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Dynamic  Relationships 


Project  Performance  Process  Capability 
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Soe  how  do  we  do  this? 


A  ML  2  Managed 

T  Project  process  capability  begins  to  mature 


Instill  (Lean)  project  performance  goals  early  -  may  be  out  of  process  scope 


A  ML  3  Defined 

i  Organizational  processes  are  defined  (Lean  as  well?) 

i  Relationship  between  organizational  and  project  processes  is  first 
established 


Ensure  the  organization  enables  lean  projects  to  perform  i  all  in  scope 


A  ML  4  Quantitatively  Managed 

T  Statistically  stable  processes 

T  Process-performance  baselines  and  models  are  established  at 
organizational  level  (OPP) 

i  Process-performance  is  managed  at  project  level  (QPM) 


Statistical  control  of  processes  and  coupling  to  process-performance 


A  ML  5  Optimizing 

T  Process-performance  is  continually  improved 


Optimizing  i  boundless  opportunities  to  improve  process-performance 
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What  does  Lean/CMMI  integration  do  for  us? 


A 

A 


Puts  process  capability  and  project  performance  in  the 
same  space/time  continuum  (at  ML1) 

V  Forces  us  to  pay  attention  to  both  at  the  same  time 

V  Before  process-performance  considerations  at  CL/ML  4  and  5 

Allows  us  to  relate  process  capability  and  project 
performance  BEFORE  CL/ML  4  and  5 

i  Which  means  we  deal  with  process  architecture  issues  much 
earlier 

T  Which  means  our  processes  perform  better  earlier  (heuristically) 

T  Which  means  we  pay  attention  to  project  PERFORMANCE  from 
Day  1  (as  a  driver?...) 
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Summary 


A  Lean  supports  integration  of  project  performance 
considerations  at  ML1,  ML2,  and  ML3 

i‘  Focus  and  priorities  for  process  improvement  effort 

T  Includes  product  quality  as  well  as  project 
performance 

A  Lean  provides  direction  for  process-performance 
models  and  baselines 

T  Lean  organization  will  help  projects  perform  better 

T  Lean  organization  will  help  projects  produce  higher 
quality  products 
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Jeffrey  L.  Dutton 

Chief  Engineer,  ITSS 

1525  Perimeter  Parkway,  Suite  510 

Huntsville,  AL  35806 

256.971.5527 

ieff.dutton@iacobs.com 


JACOBS 


17 


omplete 


Your  complimentary 
use  period  has  ended. 
Thank  you  for  using 
PDF  Complete. 


res 


ng  My  Organization  for  an 


Appraisal,  but  I'm  Not  Really  Sure 
I  Understand  this  PHD  Thing. 


Should  I  worry? 


n 


Sam  Fogle 
ACE  Guides,  LLC 


IV*  PDF 

*  Complete 


Your  complimentary 
use  period  has  ended. 
Thank  you  for  using 
PDF  Complete. 


and  Expanded  Features 


YES!!!!!! 


2 


Complete 


Your  complimentary 
use  period  has  ended. 
Thank  you  for  using 
PDF  Complete. 


jres 


•  Why? 

•  What? 

•  How? 
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□  Secure  Lead  Appraiser 

□  Confirm  Other  Team  Members 

□  Reserve  Rooms 

□  ... 

□  Arrange  for  Snacks 

□  Create  PHD 

□  Get  Flip  Chart  and  Markers 
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ptions 


•  "Creating  a  PHD  looks  straightforward,  I'll 
have  our  summer  intern  do  it  the  week 
before  the  Readiness  Review." 


•  "This  is  just  a  guide  anyway.  Even  if  it  is 
wrong,  it  is  the  appraisal  team's  duty  to 
find  the  right  stuff." 

•  "If  2  pieces  of  evidence  per  practice  is 
good,  10  pieces  should  be  5  times  as 
good." 
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Discovery  requires  that  the  team  do  a  search 
for  evidence  for  each  practice 

SCAMPI  pilots  took  more  than  a  month 

Process  needed  to  be  streamlined 
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Accuracy 


Time  (Cost) 
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3.  Verification 


Mapping 

(PHD) 


Verification  has  the  organization  provide  a 
map  that  indicates  what  evidence  is 
appropriate  for  each  practice 


The  appraisal  team  reviews  the  evidence 
and  only  needs  to  search  when  the  data 
provided  is  not  clear  or  convincing 
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*Strongly  recommended  when  not  prevented  by  security/access  issues 
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Project  Planning 
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Status 

Key  Practice  /  Notes 

Source 
of  OE 

Document  (s) 

Comments 

ORG 

J 

0 

"S' 

o 

Project  2 

j  Project  3 

Project  4 

|  Direct 

Indirect 

|  Affirmation 

Information 

Needed 

Rating 
(FI.  LI. 
PI.  Nl. 
NY) 

4 

SGI 

Estimates  of  project  planning  parameters  are  established  and 
maintained. 

5 

SP1.1 

fEstabiisft  a  top  -/eve/  work  breakdown  structure  {WBS)  to  estimate  tbe 
scope  oftfte  project. 

6 

Appraisal  Considerations: 

-  Determination  of  VBS  usage  for  this  practice  must  be  based  on  top-level  VBS  only,  not  its 
fully  elaborated  and  expanded  form  as  referenced  in  subsequent  practices  of  this  PA 

-  Top-level  work  breakdown  structure  should  be  driven  by  and  linked  to  specified  product 
requirements.  (See  Requirements  Management  PA) 

-  See  Supplier  Agreement  Management  process  area  for  more  information  about  acquiring  work 
products  from  other  sources  eternal  to  the  project 

-  Level  of  supporting  documentary  evidence  will  vary  based  on  project  size/duration.  Larger 
projects  may  have  minutes  from  estimation  meetings,  estimation  teams,  and  tools  use,  etc. 
Smaller  may  have  none.  Appraisal  team  will  need  consensus  on  the  VBS  elements  that  will  be 
expected.  See  PR  SP1.4-1  for  derivation  of  detailed  work  breakdown  structures  from  top-level 
work  breakdown  structures. 

7 

Direct  Artifact  Example: 

-  Vork  Breakdown  Structure 

-  Top-level  VBS  revision  history 

S 

Indirect  Artifact  Example: 

-  Task  descriptions 

-  Vork  product  descriptions 

-  Product  requirements,  product  roadmaps 

-  Organizational  standard  VBS  template 

-  Identification  of  work  products  for  components  of  work  products!  that  will  externally  acquired. 

3 
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•  Direct  Evidence  -  needed  for  every 
practice 


•  Indirect  Evidence  -  not  necessarily  for 
EVERY  practice* 

•  Quiz  1:  What  type  of  evidence  is  a  set  of 
meeting  minutes?  It  Depends 

•  Quiz  2:  How  many  do  you  need  of  each 
type?  it  Depends 


*Talk  to  your  Lead  Appraiser 
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•  Enough  to  convince  the  team  that  a 
practice  has  been  fully  implemented* 


•  But  NOT  everything  you  can  think  of 


*Talk  to  your  lead  appraiser 
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What  kind  of  knowledge  is  needed? 

•  Understanding  of  the  CMMI  practices 

•  Understanding  of  the  SCAMPI  method 

•  Understanding  of  how  the  project's  data  is 
organized 


Do  you  have  one  person  with  all  3? 
If  not,  then  you  need  a  team 
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•  Does  your  Lead  Appraiser  have  a  tool  that 
they  recommend? 

•  Will  you  use  the  data  for  internal  purposes? 

•  How  easy  is  it  to  put  data  in  and  get  it  out? 

•  How  easy  is  it  to  correct/update  the  data? 


omplete 


Your  complimentary 
use  period  has  ended. 
Thank  you  for  using 
PDF  Complete. 


ips 


Involve  team  members  in  quality  checks 

Don't  wait  for  Readiness  Review  to  check  the 
quality  of  the  PHD 


Evolve  PHD  over  a  series  of  appraisals 


•  Effort  devoted  to  PHD  (including  quality 
checks)  should  be  proportional  to  the 
importance  of  achieving  the  ratings 


Remember:  having  an  inaccurate  PHD  does  not  just 
make  it  harder  to  find  the  correct  data,  it  may  convince 
the  team  that  appropriate  data  does  NOT  exist 
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•  PHD  development  can  be  the  major  driver  of 
total  appraisal  cost 

•  Data  collected  over  a  set  of  appraisals  where 
the  organization  tracked  internal  effort, 
showed  PHD  preparation  to  be  ~60% 

•  Total  PHD  effort  (development,  quality, 
rework,  quality,  ...)  can  run  to  ~40  hours  per 
PA  per  project 

•  Level  2,  3  projects  =>  5  person  months 

•  Level  3,  3  projects  =>  almost  a  person  year 


omplete 


Your  complimentary 
use  period  has  ended. 
Thank  you  for  using 
PDF  Complete. 


Work  Saver 


Is  standard  process  detailed  enough  so  that  it 
will  be  fairly  consistent  from  project  to 
project  where  a  specific  type  of  data  will  be 
found? 


Provide  the  projects  with  a  PHD  that  already 
tells  them  where  to  find  the  evidence. 

e.g. 

PP  SP  2.2  Identify  Project  Risks 

see  section  3.4  of  the  project  Risk  Management  Plan 
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If  done  poorly 

•  Can  consume  vast  resources  to  prepare 

•  Will  reflect  a  poor  understanding  of  what  is  needed 

•  Will  cause  appraisal  to  proceed  very  slowly 

•  Can  confuse  the  true  state  of  the  practice 


If  done  well 

•  Will  require  limited  restarts  or  rework 

•  Accurately  reflects  the  work  done  in  the  organization 

•  Provides  an  efficient  means  for  an  appraisal  team  to  find 
appropriate  evidence 

•  Identifies  appraisal  risks  by  uncovering  holes  in 
implementation 
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•  Sam  Fogle 

•  ACE  Guides,  LLC 

•  www.ACEguides.com 

•  sam@ACEquides.com 

•  301-345-7121 
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chnology  Conference  and  Users  Group 
November  2007 


How  to  Kick  Start  a  Process  Improvement  Project 

to  Achieve  a  CMMI  Rating 

Brenda  F.  Hall 

Computer  Sciences  Corporation  (CSC) 


EXPERIENCE.  RESULTS. 


Background: 

A  A  decision  has  been  made  by  the  Powers  that  Be  that  a  particular  program 
should  "get  a  CMMI  level  3  ratingn 

A  A  search  is  made  for  a  Process  Engineer  to  lead  this  effort  and  make  this  happen 
A  You  get  the  assignment 
A  Now  what? 

A  The  purpose  of  this  presentation  is  to  guide  you  on  how  to  kick  start  this 


effort 

A  Process  Engineer: 


START  YOUR  ENGINE 
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EXPERIENCE.  RESULTS. 


AYour  first  task  is  to  determine  what  it  means,  exactly,  to  "get  a 
CMMI  level  3  rating". 

AAs  the  Process  Engineer,  not  only  must  you  decipher  the  goal  and 
rephrase  it  into  laymen's  terms  that  are  actionable  and 
achievable,  but  you  must  also  determine  what  needs  to  be  done  to 
satisfy  this  goal  in  terms  of : 

i  Resources,  including  Consulting  Support  from  Corporate  Management 
i  Teams,  Boards,  Steering  Committee 
i  Plans,  schedules,  procedures 
i  Baselines,  databases,  tools,  working  aids 
i  Training 

AThat's  just  to  name  a  few  basics  for  getting  started 
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EXPERIENCE.  RESULTS. 


ABut,  the  first  question  that  must  be  answered  with  a  resounding 


YES"  is: 


Do  you  have  Executive  Sponsorship ? 

ASenior  Management  must  sponsor  and  champion  this  cause  or 
there  is  no  way  it  can  be  successful. 

ASo,  if  the  answer  is  NOT  Yes,  then  you  really  do  not  have 
authorization  to  proceed,  nor  should  you. 

ABut,  if  the  answer  IS  Yes,  you  have  a  lot  of  work  to  do. 

ALet's  examine  the  activities,  tasks  and  support  that  must  be 
initiated  to  kick  start  this  effort  and 
institute  an  infrastructure 

for  implementing  a  process  improvement  project 


to  achieve  a  CMMI  rating. 
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EXPERIENCE.  RESULTS. 


AEstablish  a  Process  Team  to  Facilitate  the  Process  Improvement 
Project 

T  Assign  the  Process  Engineer  as  Chairperson 
i  Assign  Process  Team  Members 
A  Subject  Matter  Experts 
A  Support  Organizations 
A  Relevant  Stakeholders 

ADevelop  a  Project  Schedule 
AEstablish  a  Measurement  Program 
AEstablish  the  Process  Improvement  Infrastructure 
i  Process  Baseline 
i  Change  Request  Systems 
i  Measurement  Repository 
T  Meeting  Minutes  and  Action  Items 
T  Corrective  and  Preventive  Actions 
i  Practice  Implementation  Indicator  Descriptions  (PI IDs) 
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A  Establish  Governance 
T  Program  Directives 
i  Process  Improvement  Plan  and  Schedule 
i  Steering  Committee 
T  Configuration  Control  Board  (CCB) 

A  Align  Process  Improvement  Activities  with  Parent  Organization 
and  Corporate  Organization 

A  Formally  Introduce  the  Process  Improvement  Project  and  Team 
to  the  entire  program 


6 


>PDF 

t  Complete 


use  period  has  ended. 
Thank  you  for  using 


Your  complimentary 


PDF  Complete. 


trade  to 

and  Expanded  Features 


EXPERIENCE.  RESULTS. 


AEnsure  Process  Team  members  are  trained  in: 
i  the  CM  Ml  i  formal,  instructor-led,  3-day  class 
i  Process  Architecture  and  Modeling 
i  Process  Concepts 
A  Institutionalization 
A  Process  Maturity 
A  Process  Capability 
A  Process  Performance 

AEnsure  Program  Leadership  receives  a  CMMI  overview 
AEnsure  Program  Staff  Members  receive  a  CMMI  in-depth  briefing 
AAddress  Process  issues: 

T  How  do  we  begin  the  process  improvement  effort  with  our  current 
process  assets? 

i  What  measures  are  needed  in  addition  to  the  ones  we  are  currently 
collecting,  analyzing  and  reporting? 

T  What  information  exists  throughout  the  company  that  we  can  leverage? 
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AResolve  Interpretation  Issues 

T  Determine  the  organizational  elements,  processes  and  practices  that 
will  be  in  scope 

T  Determine  ’’alternate  practices” 

AApply  the  CMMI 

i  Conduct  a  gap  analysis  (or  a  series  of  gap  analyses)  of  the  CMMI 
against  existing  processes  and  practices 

i  Develop  action  plans  for  closing  gaps 
Almplement  the  PIP  and  the  action  plans 
APopulate  the  PI  IDs  with  evidence 
APrepare  for  appraisals 
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Interpretation  Issues  -  Scope 

AWhat  element(s)  of  the  organization  can  be  considered  a  “project”? 

T  Develop  a  Program  Profile  document  that  contains  information  about  all 
projects. 

i  Determine  from  the  profile  which  elements  of  the  organization  can  be 
considered  a  project 

i  Based  on  the  work  being  performed,  security  issues,  etc.  select  the  candidate 
projects  and  any  other  participating  elements  (e.g.,  the  PMO). 

ABased  on  the  work  performed  on  the  program: 

i  What  processes  are  being  followed?  Customer  processes? 

Your  Company  processes?  Subcontractor  Company  processes? 

This  will  determine  what  processes  are  in  scope  and  what  processes  are  out  of 
scope 
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Interpretation  Issues  -  Scope 

A  Requirements  Issues 

T  What  are  requirements? 

i  What  constitutes  a  Requirements  Traceability  Matrix? 

T  What  evidence  exists  to  show  bidirectional  traceability  of  requirements? 

A  Planning  Issues 

i  How  do  we  estimate  size,  effort,  cost  and  schedule? 

i  Are  the  methods  in  which  risks  are  being  identified,  analyzed  and  tracked 
sufficient? 

T  Do  we  develop  a  WBS  for  each  organizational  element? 

T  If  not,  what  constitutes  a  WBS? 

A  Supplier  Agreement/Procurement  Issues 

i  Does  the  SAM  PA  apply? 

i  If  so,  based  on  the  work  performed  on  the  program,  is  the  procurement  of 
labor  in  scope  (purchasing  labor  services)? 

A  What  practices  may  be  considered  (alternate  practices”? 
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AConduct  a  gap  analysis  of  the  CMMI  against  existing  processes  and 
practices 


ADevelop  action  (get  well)  plans  for 
closing  gaps 

i  Build  the  actions  into  the  Project  Schedule 
i  Monitor  and  track  the  completion  of  all 
i  actions 

AManage  the  implementation  of  the  PIP 
T  Process  Team-manage  the  activities  with 
the  guidance  of  the  Steering  Committee 
T  CCB-review  and  disposition  changes 
to  the  process  baseline 


APopulate  the  PIIDs  with  evidence 
APrepare  for  appraisals 
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AExecutive  Sponsorship 

ADedicated,  CMMI  trained  Process  Improvement  Project  Staff 
AProcess  Baseline 

ARegular,  frequent  communication  about  progress  and  needs 
i  Process  Team  Meetings 
T  Steering  Committee  Meetings 
i  CCB  Meetings 
T  All  Hands  Meetings 
i  Status  Reports 

i  Newsletters,  Posters,  Flyers,  Broadcast  email  messages,  mementos, 
etc. 

ARecognition  and  Reward  Program 

AFeedback  on  progress  (internal  reviews,  SCAMPI  C,  SCAMPI  B) 
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AEngage  the  SCAMPI  Lead  Appraiser  in  the  early  planning  stages 
and  continue  communication  until  the  rating  is  achieved 

AEnsure  that  your  organization  has  a  representative  on  the  SCAMPI 
Team 

ABegin  preparation  for  evaluations  at  least  two  months  in  advance 
and  manage  the  logistics 

AEnsure  that  Senior  Management  is  aware  of  any  barriers  or 
obstacles 
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EXPERIENCE.  RESULTS. 


AAs  a  Process  Engineer  responsible  for  preparing  a  program  to 
achieve  a  CMMI  rating,  it  is  important  that  you  realize  that  this 
effort  may  take  18-24  months  of  project  activities. 

AThe  guidelines  in  this  presentation  should  help  you  get  started. 


AQuestions???? 
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■  Overview  of  the  PIER 

■  Experiences  and  Lessons  Learned 

■  Outlook  for  the  future 
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Most  DoD  contractors  claim  high  Maturity  Levels  (Level  3 
and  above)  as  measured  by  the  Capability  Maturity  Model 
Integration  (CMMI),  yet  performance  of  individual  projects 
does  not  reflect  that  maturity. 

■  How  can  the  Government  leverage  the  CMMI  to  close  the 
performance  gaps  on  their  programs? 
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rganizations 


■  Maturity  Levels  are  indicators 
of  organizational  potential 
performance. 

■  They  describe  how  the  next 
project  may  perform  based 
on  a  sampling  of  existing 
projects. 

■  Maturity  Levels  reside  at  the 
organizational  level  and  are 
not  an  indication  of  how  an 
individual  project  is 
performing . 

■  Project  instances  may  be 
situated  in  a  different  time 
frame  and  in  a  different  part 
of  the  organization. 
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■  Original  Problem  Statement 
— ^  ■  Overview  of  the  PIER 

■  Experiences  and  Lessons  Learned 

■  Outlook  for  the  future 
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troduced  in  2004  for  ESC 


Process  In-Execution  Review  (PIER) 

Adapts  the  SCAMPI  B/C  method  for  assessing  development 
contractor  process  performance  during  source  selection  and 
contract  execution 

Applies  to  specific  projects  versus  organizational  level 

Exposes  risks  to  project  execution  -  early  or  “leading” 
indicator 

T  Tailored  to  focus  on  process  areas  of  most  interest 

Considers  the  appropriateness  of  the  process  for  the  program 

■  Our  observation  is  that  this  is  a  major  difference  between 
SCAMPI  and  PIER 


■  Goals 

i  Execute  projects  at  higher  maturity  levels 

Improve  overall  cost,  schedule,  and  technical  performance 

Tie  process  improvement  goals  and  accomplishments  to 
earned  value  and  award  fee  to  reinforce  desired  behaviors 
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>essment  Needs 


SCAMPI  Method  A 
Institutionalization 
Organizational  focus 
Rigorous,  expensive 
Ratings 

SCAMPI  Method  B 

Deployment  and  execution 
Evidence  of  implementation 
What  they  are  doing 

SCAMPI  Method  C 
Approach 
Plan  for  execution 


$ Resource  intensive 
^Limited  utility  for  full  and 
open  source  selection 


VfContract  monitoring 

/ Competitive  downselect 
-  / Limited  utility  for  full  and 

open  source  selection 

/ Contract  Monitoring 
PFuii  and  open  source 
selection 


What  they  will  do 
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■  Assess  risks  associated  with  process  development/tailoring 
and  execution  pre-contract  award 

■  Assess  risks  associated  with  process  tailoring,  execution, 
adherence,  and  capability  during  contract  performance 

T  Select  process  areas  relevant  to  project’s  timing  and  activities 

T  Assess  process  appropriateness 

■  Follow  SCAMPI  Methodology 

i  Interview  questions  based  on  model  tailored  as  appropriate 

i  Artifact  examination  based  on  performance,  quality,  specific 
program  process  requirements,  and  risk 

T  Observe  strict  confidentiality  -  non-attribution 

i  Team,  contractor,  sponsor  only 

■  Results  in  actionable  findings  by  Program  Office  and/or 
Contractor 

■  Approach  PIER  collaboratively  whenever  possible 
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■  Original  Problem  Statement 

■  Overview  of  the  PIER 

— ■  Experiences  and  Lessons  Learned 

■  Outlook  for  the  future 
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Problems  Found  During 


■  Contractor  proposed  tailoring  for  specific  processes 

I  Government  later  discovered  that  tailoring  meant  “tailor  out” 
processes  that  were  appropriate  and  applicable  for  the  program 
(e.g.,  configuration  management) 

i  Project  specific  plans  and  processes  not  developed  when 
needed;  out  of  date  or  boiler-plate  content  obviously  not  used 

■  In  depth  look  into  execution  of  processes,  evidence 
provided  by  program  artifacts,  and  staff  interviews  tell  a 
different  story  than  is  often  represented  in  management 
reviews 

I  Need  to  check  up  on  corrective  actions 

■  Software  development  processes  inadequate 

I  Lack  consideration  of  program  specific  risks  especially  for 
software  assurance 

I  Lack  firmware  development  plans  especially  for  programmable 
devices 
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Problems  Found  During 


MibKs  (continued) 


■  Product  quality  and  completeness  is  secondary  to  “on 
time”  delivery 

I  Management  unresponsive  to  quality  reports  and  audits 
especially  during  development  activities 

■  Development  configuration  control  is  often  inadequate  or 
non  existent 

i  Software  builds  especially  before  critical  design  reviews 
I  Documentation 

I  Work  products  driving  the  design  not  reviewed,  signed,  final  or 
under  configuration  control 

■  Inadequate  risk  management  programs 

I  Abandon  process 

I  Not  full  team  participation/filtered  risks 

■  Contractor  not  taking  benefit  of  lessons  learned  from  other 
programs 

■  Inadequate  stakeholder  planning  and  involvement 
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Problems  Found  During 
KitKs  (concluded) 


■  Program  information  and  materials  at  risk 

I  Security  and  back  up  systems 

■  Not  getting  the  best  out  of  engineering  tools 

i  Not  implemented  well  or  at  appropriate  point  in  the  program 
i  Personnel  not  sufficiently  trained  in  use 

■  Critical  trade  studies,  design  decisions  and  rationale  not 
documented  or  explained  in  accordance  with  documented 
organizational  processes 

■  Inadequate  or  ineffective  program  planning  and  control 

I  Not  recording  all  time  worked  -  skews  historical  information  for 
estimation 

I  Booking  credit  for  incomplete  work  packages  -  roll  problems 
forward 

I  Schedule  planning  results  in  periods  of  substantial  contractor 
and  Government  overload  for  reviews  and  meetings 

I  Inadequate  staff  planning  results  in  critical  shortfalls 
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PIER  Process 


■  Variability  of  process  execution  and  performance  varies 
widely 

i  From  contractor  to  contractor 
I  Among  contractor  teams 

i  For  different  contractor  operating  locations  or  programs 

■  Performance  is  directly  related  to  process  execution 

I  Periodic  checks  on  contractor  increases  probability  of  good 
process  execution  on  individual  programs 

■  Conduct  of  PIERs  provides  insight  not  otherwise  available 
to  Government  Program  Manager 

I  Conduct  when  artifacts  are  available  and  time  exists  to  correct 
identified  risks 

■  Between  requirements  review  and  design  review 

■  Plan  PIERs  so  as  to  minimize  program  disruption  and 
maximize  participation 

I  Between  System  Requirements  Review  and  Design  Reviews 
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PIER  Process 


■  Program  Manager  must  act  on  the  information  resulting 
from  the  PIER  in  a  timely  manner 

I  Capture  observations  and  trends  to  isolate  potential  systemic 
problems 

I  Improvements  required  for  contractor  and  Government 

■  PIER  teams  should  be  led  by  individuals  with  CMMI, 
SCAMPI,  technical,  and  program  management  background 

I  Especially  important  to  have  some  knowledge  of  the  program 
and  topic  area 

I  Certified  SCAMPI  B/C  Team  Lead  or  Lead  Appraiser  PIER  teams 
should  have  a  mix  of  technical  backgrounds  relevant  to 
program 

I  One  contractor  team  member  from  outside  the  program  (and 
preferably  the  organization) 

■  Consider  the  type  of  program,  the  stage  of  development, 
and  asserted  organizational  maturity  level  in  selecting 
process  areas 
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PIER  Process 


■  Contract  execution  PIERs  for  process  improvement 

T  Independent  Team  Lead 

I  Mix  of  contractor  and  program  office  team  in  collaborative 
environment 

I  At  least  one  team  member  independent  of  program 
I  Team  training  to  include  site  coordinator 

■  Contract  execution  PIERs  for  Award  Fee 

I  Government  team 

■  Acquisition  organization  must  have  a  consistent  approach 
to  conducting  PIERs 

I  Need  guidance,  templates,  and  training  to  ensure  consistency 
of  PIERs 


■  Government  needs  a  method  for  collecting  PIER  results 
(non-attribution)  to  isolate  systemic  problems  in 
acquisitions 

■  PIERs  are  mentally  and  physically  challenging  but  worth  the 
effort 
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■  Original  Problem  Statement 

■  Overview  of  the  PIER 

■  Experiences  and  Lessons  Learned 
— ^  ■  Outlook  for  the  future 
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:he  Future 


■  Add  financial  management  and  Cost  Account  Management 

i  Assess  execution  of  Earned  Value  Management  System  (EVMS) 
practices  especially  in  correlating  product  maturity  and 
performance  to  earned  value 

I  May  integrate  with  the  Integrated  Baseline  Review  (IBR) 
process 

■  Deeper  look  into  product  quality 

I  Technical  maturity  and  product  performance  using  models, 
simulations,  prototypes,  and  early  functional  assessments 

I  Identification  and  implementation  of  Technical  Performance 
Measures 

■  Modify  PIER  for  CMMI  vl.2  for  Development 

I  Adjustments  based  on  CMMI-ACQ  when  available 

I  Adjustments  based  on  CMMI-SVC  when  available 

■  Apply  PIERs  to  Government  as  well  as  Government/ 
contractor  teams 
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■  ESC  has  conducted  about  18  PIERS  for  various  programs 

■  The  SCAMPI-based  PIER  provides  valuable  insight  into 
contractor  capability  on  a  project-by-project  basis, 
supplementing  technical  activities,  and  providing  a  basis 
for  risk  assessment,  performance  feedback,  incentive 
management,  and  program  office  commitment 

■  The  PIER  is  gaining  acceptance  in  the  acquisition 
community,  being  integrated  into  past  and  present  program 
plans 

■  Planning  for  future  PIERS  will  leverage  current  lessons,  and 
will  adapt  as  the  CMMI  changes 


We're  watching  you... 
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ormation,  Contact: 


■  Dale  Swanson 
The  MITRE  Corporation 
202  Burlington  Road 
Bedford,  M A  01730-1420 
781-266-9195 
swaneed@mitre.org 


■  Lynda  Rosa 
ESC  ACE  Chief  Engineer 
9  Eglin  Street 

Hanscom  AFB,  MA  01 731  -21 00 
781-377-5398 

linda.rosa.ffrdc@hanscom.af.mil 

lmrosa@mitre.org 


■  Jennifer  Hoberman 
The  MITRE  Corporation 
202  Burlington  Road 
Bedford,  M A  01730-1420 
781-266-9581 
ikh@mitre.org 
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SEI  Appraisal  System: 

Expectations  for  Usage 
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Required  Use  of  SEI  Appraisal  System 


Recent  Policy  Announcement 

•  All  SCAMPI  appraisals  must  be  setup  in  advance  of  Phase  II  (Onsite) 

—  Specified  items  from  Appraisal  Input  30  days  in  advance 
—  Remaining  items  from  Appraisal  Input  7  days  in  advance 
—  All  elements  of  the  Appraisal  Plan  1  day  in  advance 

•  Business  cycles  are  sometimes  shorter  than  30  days 

—  Exceptions  will  be  handled  routinely 

o  These  are  not  waivers,  but  expected  occurrences 
o  Exceptions  are  not  intended  to  be  the  norm 
—  Discussing  how  to  handle  this  within  SAS,  rather  than  email. 
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Required  Use  of  SEI  Appraisal  System2 


Constructive  Feedback  We’ve  Received 

•  Concern  about  limited  value  for  requiring  B&C  setup  in  advance 

—  We  agree,  this  does  seem  excessive  -  we  will  change  it 

•  Wording  of  the  communication  is  needlessly  overly  complex 

— "...  no  later  than  30  days  prior  to  the  start  of ...” 

—  Simpler  to  say  “. . .  at  least  30  days  prior ...” 

•  The  policy  does  not  address  closure  of  SAS  records  in  a  timely  way 

—  Contract  language  says  all  data  must  be  submitted  within  30  days 

o  We  will  require  complete  SAS  record  within  30  days  of  the  end 
of  phase  II  (onsite) 
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Required  Use  of  SEI  Appraisal  System3 


What  About  SEI  Response  Time? 

•  We  are  committed  to  make  a  determination  within  30  days 

—  The  quality  review  clock  starts  when  the  LA  has  successfully  hit  the 
“submit  button”  -  all  required  fields  are  complete 

—  Within  30  days  we  will  report  either: 

o  The  SAS  record  has  passed  our  quality  review 

o  There  is  an  issue  that  requires  attention  from  the  LA 

—  The  routine  things  take  less  than  30  days 

o  Appraisal  program  manager  is  tracking  this  cycle  time 

•  Most  LAs  are  very  timely  in  their  responses  to  our  follow-ups 
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High  Maturity  Appraisals: 

Certification,  Reporting  and  Auditing 
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Certification 


A  Maturing  Profession 

•  We  want  to  protect  your  investment,  and  the  investment  of  many  others 

•  Raising  the  floor,  not  pushing  the  envelope 

—  Base  all  appraisals  outcomes  on  common  criteria 
—  Assure  that  organizations  realize  intended  benefits 
—  Improving  the  engineering  practice  is  our  mission 
Confluence  of  Many  Stakeholder  Voices 

•  SEI  is  balancing  many  legitimate  inputs 

—  Optimal  technical  solutions  are  not  always  obvious  to  everyone 
—  We  are  working  to  be  inclusive,  and  many  have  contributed 
—  Valuable  advice  and  technical  input  received  at  every  step 
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Reporting  High  Maturity  Appraisals 


Increase  Confidence  in  Reported  Results 

•  Requiring  supplemental  information  increases  transparency 

•  Transparency  serves  people  operating  in  good  faith  the  most 

Compliance  to  Reporting  Requirements  Has  Been  an  Issue 

•  Not  every  LA  seems  to  be  aware  of  the  specific  requirements 

•  Motivation  behind  requirements  may  not  be  obvious  to  some 


We  are  working  on  this  to  make  it  easier  for  all  parties, 
and  to  better  achieve  our  objective. 
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Reporting  High  Maturity  Appraisals2 


The  Stated  Requirements: 

SCAMPI  Method  Definition  Document  -  Page  11-119  (3.1.1  Parameters  &  Limits) 

If  the  appraisal  resulted  in  capability  or  maturity  level  4  or  5  ratings,  the 
organizational  processes  or  subprocesses  and  the  corresponding  PAs 
that  were  determined  to  be  under  statistical  control  must  be  included  in 
the  final  findings. 

SCAMPI  Method  Definition  Document  -  Page  11-130  (3.2.2  Parameters  &  Limits) 

For  high  maturity  appraisals  (capability  or  maturity  level  4  or  5),  the 
appraisal  team  leader  must  validate  that  a  substantial  portion  of  the 
organization’s  and  projects’  quality  and  process-performance  objectives 
and  statistically  managed  subprocesses  can  be  mapped  directly  to  and 
support:  (1)  the  established  business  objectives  as  stated  and 
disseminated  to  key  employees  of  the  organization,  and  (2)  the  needs 
and  priorities  of  customers,  end  users,  and  other  stakeholders.  This 
validation  is  required  to  prevent  the  granting  of  high  maturity  ratings  for 
trivial  improvements. 
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Reporting  High  Maturity  Appraisals3 


Appendix  A:  Appraisal  Disclosure  Statement  Template: 

Additional  Information  for  Appraisals  Resulting  in  Capability  or 
Maturity  Level  4  or  5  Ratings 

Describe  which  processes  or  subprocesses  are  under  statistical 
management  and  were  included  in  the  objective  evidence  for  this 
appraisal.  Also  list  the  PA(s)  and  organizational  quality  and  process- 
performance  objective(s)  these  processes  or  subprocesses  pertain  to. 


Process/Subprocess 

Process  Area(s) 

Quality/Process  Performance  Objective(s) 
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Reporting  High  Maturity  Appraisals4 


SAS  User’s  Guide,  Page  1 1 : 

Beneath  the  model  display,  you  will  see  fields  to  enter  the  Assigned  Maturity  Level.  If  you  are 
assigning  a  Maturity  Level  4  or  5,  there  is  a  comment  box  provided  for  your  description  of  the 
critical  subprocesses  under  statistical  management  that  have  earned  this  rating. 

Describe  which  processes  or  subprocesses  are  under  statistical  management  and  were 
included  in  the  objective  evidence  for  this  appraisal.  Also  list  the  PA(s)  and  organizational 
quality  and  process-performance  objective(s)  these  processes  or  subprocesses  pertain  to. 

An  acceptable  response  in  this  field  will  identify  the  following: 

■  The  process/subprocess  being  statistically  managed 

■  The  process  area(s)  quality/process 

■  The  performance  objectives  being  met 

*  QUALITY  NOTE:  *  The  data  input  into  the  SAS  record  will  be  displayed  exactly  as  entered  on 
the  ADS  and  on  the  PARS  website.  Entering  references  to  the  Plan  or  other  submitted  artifacts 
will  not  present  your  appraisal  data  in  a  positive  way.  It  will  be  questioned  during  our  Quality 
review  and  will  require  the  appraisal  to  be  reopened  for  you  to  correct  the  data. 
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Reporting  High  Maturity  Appraisals5 


Three  Items  of  Information  Are  Required 

•  Subprocess 

•  Process  area 

•  Goal  served  through  statistical  management  of  the  subprocesses 

Serving  Business  Goals  is  the  Most  Important  Thing 

•  The  motivation  for  asking  is  to  assure  that  ‘trivial’  applications  of 
statistical  techniques  are  not  accepted  as  evidence 

—  The  intent  is  to  rigorously  manage  the  things  that  have  impact 

—  Level  4  is  not  about  analysis  of  large  amounts  of  data 

—  Level  5  is  not  about  ‘one-off  six  sigma  projects 
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Reporting  High  Maturity  Appraisals5 


Common  Issues  Encountered: 

•  No  mention  of  subprocesses 

•  No  mention  of  business  goals 

•  No  mention  of  process  areas 

•  Very  high-level  goals  mentioned 

•  Providing  just  a  list  of  measures  or  analyses 

•  Very  large  amount  of  information,  often  relating  to  other  topics 

—  Process  Performance  Baselines 
—  Process  Performance  Models 
—  History  of  process  improvement 
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Questions  and  Answers: 

Questions  Receiving  Attention 
Answers  Being  Developed 
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Managing  Potential  Conflicts  of  Interest 


Some  Things  Are  Very  Clear 

•  “Do  not  appraise  your  own  work” 

•  Authoring  the  process  means  you  can’t  appraise  it 

Where  is  the  Boundary? 

•  How  much  coaching  do  you  have  to  do  before  you  cross  the  line? 

•  How  much  training  is  too  much  to  maintain  objectivity? 

•  Can  we  really  maintain  an  objective  view  if  we  do  a  series  of  informal 
appraisals  leading  up  to  the  benchmarking  appraisal? 

•  Is  it  acceptable  to  be  a  team  member  on  an  A  where  you’ve  consulted? 

SEI  Policy  Will  Be  Enhanced 
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On-Site  Auditing  -  Increasing  Level  of  Activity 


SEI  Auditing  Program  is  About  to  Expand 

•  We  are  hiring  new  senior  level  staff 

•  We  are  budgeting  for  more  on-site  audits 

Good  Experience  To  Date 

•  Mentoring-focused  audits  have  been  well  received 

—  Helping  Lead  Appraisers  to  see  improvement  opportunities 
—  We  were  received  in  the  spirit  of  improvement 

•  Site  visits  following  appraisals  are  very  challenging 

Audits  Will  Now  be  “Real  Time”  During  Appraisal  Events 

•  Random  as  well  as  event-driven  audits  will  be  done 

•  The  goal  is  to  assure  quality  practices  -  not  to  mentor,  per.  se. 
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Continuing  Professional  Education 


Maintaining  Authorization  and  Certification  is  Important 
•  A  maturing  profession  requires  a  focus  on  competencies 
—  SLA-BOK  work  has  been  very  well  received 
—  CLF  calls  out  a  lifecycle  of  developing  competencies 

o  This  lifecycle  does  not  end  when  certification  is  achieved 
o  Maintaining  credentials  must  be  an  active  process 
o  Developing  deeper  capabilities  are  required 

SEI  Work  Continues  Under  the 
Leadership  of  Steve  Masters 
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New  Member  of  Our  Team 


A  new  member  has  joined  the  Appraisal  Program  Quality  Team 

•  This  person  is  charged  with  helping  to  bring  continuity  and  consistency 
to  the  processes  we  use,  and  to  the  workload  we  manage. 

•  An  advocate  for  effective  communications  among  stakeholders  in  the 
process  -  SEI  staff,  as  well  as  Partners  and  Lead  Appraisers. 

•  We  are  pleased  to  welcome 

Darlene  Moore 

dmoore@sei.cmu.edu 
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3D  -  Where  We  Operate 


#  Melbourne,  FL 

#  Lisbon,  Portugal 

#  Sao  Paulo,  Brasil 

#  Hong  Kong,  China 
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ISD  -  What  We  Do 


IT  Strategy 
and 

Governance 


Appraisals 
and 
Audits 


CM  Ml,  CobiT,  Consulting 
BSC,  ITIL,  \  Coaching 

ISO9001 :2000, 

Six  Sigma,  eSCM, 
and  others 


Training 


PPQA,  PMO 
and  EPG 
Outsourcing 
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A  What  is  the  current  problem  state? 

A  What  is  the  correlation  between  business  needs  and 
improvement  frameworks? 

A  What  is  the  Integrated  System  Framework  (ISF)  and  how  will 
it  help? 

A  Next  steps?  Questions? 
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sening  Thoughts  -  “Frameworks 
lire”  -  The  quest  for  a  “single  model” 

is  lost 


A  Process  standardization  and  improvement  efforts  are  expanding  across 
the  entire  enterprise. 

T  Process  models  and  frameworks  are  proliferating  to  focus  on  different 
domains/disciplines  within  an  enterprise. 

T  The  impact  and  implementation  is  global. 

T  Compliance  requirements  levied  by  customers  using  these  frameworks  is 
driving  costs  in  the  opposite  direction  of  management  desires. 


A  Domain  and  business  area  specific  reference  models  and  frameworks 

T  Directly  address  process  needs  of  specific  sub-communities  on  both  the 
client  and  provider  sides. 

A  Can  cause  sub-optimal  investments  in  process 
A  Can  cause  counter  productive  implementations 
A  Produce  large  expense  side  inefficiencies 
T  Can  be  successfully  integrated  into  an  enterprise  improvement  effort. 
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ISD  -  Who  We  Are 


Integrated  System  Diagnostics  (ISD)  is  a  multinational  company 
dedicated  to  process  improvement,  quality  and  performance 
management. 

ISD  is  one  of  the  largest  Software  Engineering  Institute  (SEI)  Partners 
and  has  been  working  together  with  the  institute  in  researching, 
developing  and  delivering  services  (consulting,  training  and  audits) 
related  to  several  best  practice  models  (SW-CMM,  CMMI,  People 
CMM)  and  appraisal  methods  (CBA-IPI,  SCE,  SCAMPI). 

ISD  is  also  an  IT  Services  Qualification  Center  (ITSqc)  Partner  for 
delivering  services  (consulting,  training  and  audits)  related  to  eSCM-SP 
and  eSCM-CL  (IT-Enabled  Sourcing  Capability  Models). 

SEI  and  ITSqc  are  entities  of  Carnegie  Mellon  University 
SEI-  Software  Engineering  Institute 
ITSqc  -  IT  Services  Qualification  Center 
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Sponsor’s  Appraisal  Nightmare 


~\ 

Scenario  -  Organization  X 


This  organization  will  go  through: 


•  ISO9001  certified 

•  IS020000  certified 

•  CMMI  Level  3 

•  CobiT  oriented 


> 


J 


•  5  to  7  appraisals  /  audits  a  year 

•  10  to  14  appraisals  /  audits  in  2  years 

•  25  to  35  appraisals  /  audits  in  5  years 


A  Imagine  a  company  with  5  organizations  like  this  one! 
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lizational,  Strategic,  and  Operational 
Challenges  Magnify  the  Problem 


A  How  is  the  organization  defined 

T  How  to  identify/communicate  with  stakeholders 

A  Multi-national  and  Multi-geographic 
T  Norms,  culture,  and  values 
i  Languages,  time  zones,  locations 

A  Operational/Time  Constraints 
T  Business  pressure 
T  Management  pressure 
T  Stockholder  pressure 


A  Many  affected  groups 

i  Large  scope  and  risk  adds  complexity,  which  leads  to  longer  deployment. 

T  Outsourcing  impacts  more  groups,  adding  more  points  of  potential  rfailure.o 
i  Different  targeted  groups 

A  adds  to  cultural  and  legacy  complexity. 

A  exhibit  varying  levels  of  process  maturity. 


Some  slide  content  adapted  from  Paul  ByrnesoINCOSE  2000  presentation  8 
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ovement  Program  Risks  Magnify  the 

Problem 


Risk 

Factors 

Insufficient  senior 

management 

commitment 

Caused  by  turnover  or  mergers 

Based  on  disillusionment  with  results 

Resulting  from  shifting  investment  priorities 

Due  to  inadequate  resource  allocation 

Middle 

management 

resistance 

Overriding  pressure  for  project  performance;  Incentives  on  delivery, 
not  quality 

Doubt  about  seriousness  of  senior  leadership 

Inappropriate 
improvement  goals 

Level  5  in  1  year 

75  business  units  to  be  assessed  by  year  end 

Unrealistic 

expectations 

The  great  productivity  gap  related  to  managing  change 

The  technology  adoption  curve  and  change  management  awareness 
Lack  of  motivation  for  or  continuous  focus  on  process  improvement 

Crash 

implementations 

No  plans  or  long-term  perspective,  and  lack  of  following  through  on 
improvement  efforts 

Termination  of  activities  before  they  are  institutionalized 

Slide  from  Paul  Byrneso2nd  ISD  Customer  Conference  presentation 


®  2005-7,  Integrated  System  Framework  for  Excellence  or  ISF  for  Excellence  -  All  rights  reserved  -  ISD  Brasil 


I  .■-PDF 

t  Complete 


Your  complimentary 
use  period  has  ended. 
Thank  you  for  using 
PDF  Complete. 


and  Expanded  Features 


ion  Goals  -  Current  Issues 


Common 

Goal 

Sub-Goal 

Current  State,  Issues 

Ensure  results 

Contribute  directly  to 
business  improvement 

Benchmark  events  less  so  than 
interim  events 

Comparable  across 
companies/organizations 

More  so  when  externally  driven 
needs  and  teams  (benchmarks) 

Optimize  value  to 
sponsors 

Support  business  objectives 

Multiple  requirements  must  be 
satisfied 

Optimize  cost  and  minimize 
disruption 

Can  be  more  costly  without 
changes  in  approach 

Ensure  appraisal 
reliability 

Create  repeatable  processes 
i  standardize 

Too  many  improvement 
frameworks?? 

Make  results  predictable  and 
differences  explainable 

Benchmarks  (if  any)  not  standard 

Results  independent  of  team 
composition 

Objectivity  an  issue  for  both 
outsourcer  and  service  provider 

Slide  adapted  and  updated  from  presentations  by  Mr.  Byrnes  while  managing  the  appraisal  project  at  the  SEI. 
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#  Address  a  global,  systemic  enterprise  problem  of 
implementing,  managing,  maintaining,  and  complying  with 
multiple  process  models,  frameworks,  and  methods. 

#  The  Integrated  System  Framework®  provides  one  part  of  a 
technical  solution  to  client  requirements  for 

#  Optimizing  cost  to  effectively  demonstrate  ongoing  process 
adherence  to  multiple  standard  models. 

<§>  Leveraging  process  investments  across  the  enterprise  to  increase 
effectiveness  of  process  improvement  efforts. 

<§>  Increasing  synergy  across  business  areas  to  improve  process 
implementation  efficiency 

#  Contribute  to  the  professional  model  based  process 
improvement  community  and  positively  influence  its  future. 
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)verlap  of  Key  Models 


A 


A 


Most  standards/models  have  content 
overlap 


ITIL:  focus  on  IT 
Service  Management 


Six  Sigma:  focus  on  data-driven  decision 
making/improvement 


Source:  Accenture.  Used  with  permission 


T  Often  based  on  Total  Quality  Management 
(TQM)  and  Deming®  plan-do-check-act 
principles 

i  Some  core  topics  show  up  in  most  models 

Each  industry  standard/model  has  a 
sweet  spotoor  particular  area  of  focus. 

For  example: 

T  CMMI  is  particularly  focused  on  systems 
development  and  maintenance 

T  eSCM-SP  is  focused  on  IT-enabled 
sourcing 

i  COPC  is  focused  on  customer  care 

T  ITIL  is  focused  on  IT  Service  Management 
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ss  Model  Integration:  EDS  view 


. 


Project 

Driven 

Work 


Global  Knowledge  Management  Solution 

End-to-End  Delivery 

ISO 


CM  Ml  primarily  sup  ports  traditional 
software  and  engineering  projects 


Integrated  Proces 


Business 

Process 

Support 


tom  m:<  mjm 

A 


eSCM 


IHL  provides  best  practices  for  service 
management  and  service  delivery 


covers  the  entire  spectrum  of  Hie  contract  sourcing  lifecycle 


Source:  EDS.  Used  with  permission 
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eSCM-SP  CobiT  ISO  9001  BS  15000  CMMI  COPC 


Knowledge 

Management 

• 

€ 

4 

4 

€ 

c 

People 

Management 

• 

4 

4 

€ 

€ 

4 

Performance 

Management 

• 

4 

4 

4 

€ 

4 

Relationship 

Management 

• 

« 

4 

4 

C 

C 

Technology 

Management 

• 

4 

€ 

4 

c 

C 

Threat 

Management 

• 

4 

€ 

4 

c 

4 

Contracting 

• 

€ 

€ 

€ 

€ 

C 

Service  Design 
&  Deployment 

• 

€ 

4 

4 

4 

4 

Service 

Delivery 

• 

4 

4 

4 

C 

4 

Service 

Transfer 

• 

o 

€ 

O 

O 

O 

#= 

fully  £  = 

largely  0=  partially 

O 

not  covered 

Slide  courtesy  of  ITSqc  at  Carnegie  Mellon  University 
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it  help? 
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IT  Governance  Areas 


IT  strategic  alignment  and  execution  Y  know  your  business  and 
align  with  it 

IT  Performance  Management  Y  manage  your  performance  qualitative  and 
quantitatively 

Innovation  Strategic  Projects  T  select  and  manage  the  right 
projects  and  add  value  to  the  organization 


Risks  and  Operations  Y  manage  your  risks  and  operations  and  take 
preventive  and  corrective  actions  in  incidents 


Structured  and  Facts-Based  Decision  Y  take  decisions 
appropriately  (time  and  discipline) 


& 


Suppliers  and  Sourcing  i  use  the  best  balance  between  insourcing  and 
outsourcing,  and  manage  your  external  and  internal  suppliers 

Resource  Management  Y  minimize  costs  and  make  the  best  use 
of  all  resources 


Management  Process  and  Systemic  View  Y  continuously  improve  your 
value  chain  and  grow! 


People  Management  Y  cultivate,  manage  and  retain  your  talents 


®  2005-7,  Integrated  System  Framework  for  Excellence  or  ISF  for  Excellence  -  All  rights  reserved  -  ISD  Brasil 


\ 


16 


%  Complete 


Your  complimentary 
use  period  has  ended. 
Thank  you  for  using 
PDF  Complete. 


and  Expanded  Features 


>o  not  Reinvent  the  Wheel! 


IT  Governance  Architecture 


IT  Governance  themes 


IT  strategic  alignment  and  execution  T  know  your 
business  and  align  with  it 

IT  Performance  Management  i'  manage  your  performance  qualitative  and  quantita| 

Innovation  Strategic  Projects  T  select  and  manage  the 
right  projects  and  add  value  to  the  organization 

Risks  and  Operations  i  manage  your  risks  and  operations  and  take  preventive  < 
corrective  actions  in  incidents 

Structured  and  Facts-Based  Decision  T  take  decisions 
appropriately  (time  and  discipline) 

Suppliers  and  Sourcing  T  use  the  best  balance  between  insourcing  and  outsourq 
nnana/70  \/oUr  external  and  internal  suppliers 

Resource  Management  V  have  the  lower  possible  cost 
and  make  the  best  use  of  all  resources 
Management  Process  and  Systemic  View  T  continuously  improve  your 
value  chain  and  grow! 

People  Management  V  cultivate,  manage  and  retain 
your  talents 


World  of  Best  Practices 
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->r ‘Eyceffence  -  Bringing  It  Together 


IT  Governance  Themes 


IT  strategic  alignment  and  execution  T  know  your 
business  and  align  with  it 

IT  Performance  Management  T  manage  your  performance  qualitative  and  quantita| 


Innovation  Strategic  Projects  V  select  and  manage  the 
right  projects  and  add  value  to  the  organization 

Risks  and  Qper^^^y  manage  your  risks  and  operations  and  take  preventive 
corrective  actions  fn  incidents 

Structured  and  Facts-Based  Decision  T  take  decisions 
appropriately  (time  and  discipline) 

Suppliers  and  Sourcing  T  use  the  best  balance  between  insourcing  and  outsourq 
\/nUr  external  and  internal  suppliers 

Resource  Management  T  have  the  lower  possible  cost 
and  make  the  best  use  of  all  resources 


ISF  for (Eycetfence 
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^eed  to  Put  the  Pieces  Together !! 


Reduce  redundancy 
Improve  integration 
Create  synergy 
Leverage  best  practices 
Make  frameworks  transparent 


ISF  ! 
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A  What  is  the  current  problem  state? 

A  What  is  the  correlation  between  business  needs  and 
improvement  frameworks? 

A  What  is  the  Integrated  System  Framework  (ISF)  and  how 
will  it  help? 

A  Next  steps?  Questions? 
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allenge  -  How  Do  We  Integrate  All 
the  Models? 


Most  of  the  models,  frameworks,  and  best 
practices  share  a  common  set  of  principles, 
process  areas,  and  practices 


A  Senior  Management 
Commitment 

A  Leadership 

A  Costumer  Focus 

A  People  Focus 

A  Systemic  View  Focus 

A  Management  by  Process 

A  Decisions  Based  on  Facts 

A  Learning 

A  “Win-Win  ”  Partnership 
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ORGANIZATIONAL  EXCELLENCE  & 
QUALITY 
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For  each  process  "class"  and  "category"  there  will  be  an  unique 
set  of  "CPP"  (critical  process  for  the  performance)  that  will 
address  (map)  all  the  models  and  best  practices  minimizing  or 
eliminating  redundancy  and  respecting  the  overlaps. 
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for  (Excellence—  Systemic  View 


The  rgreend 
titles  are  called 
rClasseso 


Organizational 


Compi 

stitors 

r 

Leadership, 
Culture,  Strategic 
Alignment 

Market 


Infrastructure  & 
Technology 


Measurement, 
Innovation  and 
Improvement 


Customers 


Customers 

(pre-contract) 


The  boxes  are 
called  rCategorieso 


P&S  Execution 

P&S  Sustainment 

Support 

Management 


Suppliers 


* 

I 

& 

s* 


F 


Customers 


Customers 

(post-contract) 


Results  (performance) 
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SF  for  <Ej(ce  [fence  —  CPP  Examples 


Organizational 


f 

-  : 

Market  & 
Competitors 

Human 


Culture,  Strategic 


Infrastructure  & 
Technology 


'Measurement, 
Innovation  and 


Customers 


Customers 

(pre-contract) 


P&S  Execution 


D&S  Sustentatfr 


c 

Support 

~n 

v  r~ 

Management 

r 

SuDDlier^^ 

— 

§  j 

—* 

-  Customers 

Customers 

(post-contract) 

^esult^ge^nnancg^ 


Category:  Market  and  Competitors 

#  Benchmarking 

#  Brand  Management 

#  Market  Knowledge 

#  Stakeholders  Management 


Category:  Measurement,  Analysis  and  Improvemei 


Critical  Process  Performance  (rCPPsc) 
streams  are  similar  to  the  concept 
of  (process  areas. 6 


#  Measurement  and  Analysis 

#  Performance  Management 

#  Continuous  Improvement  Management 

#  Process  Assets  Management 

#  Innovation  and  Performance  Management 

#  Causal  Analysis  and  Resolution 

#  Knowledge  Management 


25 


®  2005-7,  Integrated  System  Framework  for  Excellence  or  ISF  for  Excellence  -  All  rights  reserved  -  ISD  Brasil 


I  .■-PDF 

t  Complete 


Your  complimentary 
use  period  has  ended. 
Thank  you  for  using 
PDF  Complete. 


and  Expanded  Features 


SF  for  <Ej(ce  [fence  —  CPP  Examples 


A  | 

Customers 

-> 

|  P&S  Execution 

D&S  Sustentatff5n| 

Customers 

(pre-contract) 

u 

Management 

f 

Suppliers 

— 

\  J 

^esult^ge^niianc^^ 


Critical  Process  Performance  (rCPPsc) 
streams  are  similar  to  the  concept 
of  (process  areas. 6 


Infrastructure  and 
Technology 


Capacity  Management 

#  Continuity  Management 

#  Availability  Management 

#  Security  Management 

#  Portfolio  Management 

#  Infrastructure  Management 

#  Financial  and  Cost  Management 


Support 


#  Incident  Management 

#  Problem  Management 

#  Configuration  Management 

#  Release  Management 

#  Change  Management 

#  Quality  Assurance  Management 
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for ‘Eycefl'ence  -  Automated  Tooling 
{Model  Wizard  ™) 


1 


“  Model  Elements  (MM002)  -  A  -  ISF  for  Excellence 


File  Edit  Model  Element  View  Text 


#  Staffing 

#  Competency  Man 

#  Training 

#  Career  Management 

#  Compensation 

#  Performance  Manageme 

#  Team  Management 

#  Work  Environment 

#  Mentoring  and  Coaching 


B-OJ  A  ■  ISF  for  Excellence 

s)  Views  and  Diagrams 
E)  Organizational 

EEI-{E)  Leadership,  Culture  &  Business  Alignment 
d-  (E)  Human  Capital 
H  HC  View 

Competency  Management 
Workforce  Planning 


Training  -  DEMO 


(i)  Direct  Objectives  (TRA) 

(i)  Sustaining  Objectives  [TRA] 

Q[)  Mapping  (TRA) 

■  Staffing 
i  Compensation 

■  Career  Development 
1  Team  Development 

■  Mentoring  &  Coaching 
1  People  Performance  Management 

(E)  Infrastructure  St  T echnology 

(E)  Measurement,  Innovation  and  Continuous  Improvement 

)  Relationship  St  Stakeholders 

(E)  Customer  Relationship 

(E)  Market  St  Competitors  Relationship 

(E)  Supplier  Relationship 

)  Products  St  Services 

(E)  Engineering 

(E)  Support 

(E)  Project  Management 


Element  Name: 

|T raining  ■  DEMO 
Element  Abbreviation: 

[TRA 

Element  Type: _ 

]  Critical  Process  to  Performance 

Full  Description  j  Short  Description  j  CMM  Based  Fields  j 


+  New  Element 


+  New  Child 


|  Delete 


T 


^  MoveLJp 


^  Move  Down 


The  purpose  of  "Training"  is  to  develop  people  competencies  (knowledge,  skills,  attitudes 
and  process  abilities)  aiming  at  to  create  a  learning  environment  that  possibilities: 
people  to  execute  their  activities  in  an  efficient  and  affective  manner 
people  to  take  advantage  of  career  development  opportunities 
organization  to  deploy  their  strategy  through  people  competency  development 


m  Edit  Text 


iff  Save 


Cancel 


J~|  Close 
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(Model  Mapper™) 


Model  Elements  (MMQ02)  -  A  -  ISF  for  Excellence 


Model  Element  View  Text 


X 


Map  Elements  (MM007)  -  ISFtoCMMIc 
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for <E%ceCCence  -  Automated  Tooling 
{Appraisal  Wizard™) 


W  Element  Review  <AM009)  Element:  2  Training  Needs 


Options  Filtering  Elemen^iltering  Tree  View  Record 


The  organization,  units  and  workgroups  (e.g.  project 
team)  identify  its  training  needs  from  various  sources: 
competencies,  projects  and  engagement  needs, 
customer  requirements,  strategy,  career,  etc. 


Rating  Level: 

|  Pilot  Organization 


q  b  |  XL  Close  | 


Element  Records  J  EJ^BWfl^Scuments  j  Element  Mapping  | 

|j  Save  Rec  j  ^  Spell  |  **  Delete  Rec  |  Cancel  Changes 


Record  Type 

Rec  ID  j 

[Records  [1]  +  \ 

|  Status 

Findings 

2 

The  X  organization  identifies  its  training  needs 

from  various  sources:  competencies,  projects 

and  engagement  needs,  customer 

requirements,  strategy,  career,  etc. 

- _ 

Raftird  Fields  !  Projects  Elements  /  Data  Sources ^eam  Members  |  Record  Documents  | 
Connections  |  ^  Delete  |  +  Go  to  |  [g^^ate  all  | 


Elements  Attached 


Team  Members  Data  Sources 


Model 

Element  i 

► 

ISF 

2  Training  Needs 

PCMM 

TD  G1 

PCMM 

TD  P  02 

CobiT 

P07  Manage  IT  human  resources 

CobiT 

P07.2  Personnel  Competencies 

CobiT 

P07A  Personnel  Training 

eSCM 

ppl07_b1 

ITIL 

3.3  Competence,  Awareness  and  1 

PNQ 

1  Identification  of  training  and  deve 
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AAppraisal  Method 
Almprovement  Approach 
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ISF®  T  Meta-Model 
Framework 


r  ^ 

Comprehensive 
Appraisal  Method 
(CAMsm)i  Integrated 
Appraisal  Method 

L_ _ J 


r  ^ 

Appraisal 
Wizard  E  and 
Model  WizardE  T 
Operational  Tool  Suite 

v _ J 
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System  Component 

Positioning 

The  Integrated  System 
Framework" 

#  Is  a  conceptual  vehicle  to  relate  an  organization® 
process  architecture  to  multiple  standard  models;  and 

#  Helps  to  maintain  and  measure  process  compliance 
across  multiple  models  simultaneously. 

The  Comprehensive 

Appraisal  Method  (CAM  ) 

#  Provides  a  integrated,  tailorable,  rigorous,  extensible, 
model-hneutraloappraisal  method  for  use  across  models 

#  Is  suitable  for  conducting  Process  Assurance,  Project 
Progress  Tracking,  Enterprise  Process  Oversight,  and 
Formal  Benchmark  compliance  determinations/audits. 

TM 

Appraisal  Wizard  /Model 
Wizard™  V7 

#  Provides  robust  support  for  operationalizing  the 
conceptual  framework,  and 

#  Enables  conducting  Process  Assurance  monitoring  and 
Formal  Benchmarking  compliance  activities  in  an  effective, 
efficient,  automated  manner. 

Enterprise  Process 
Improvement/Appraisal  Life 
Cycle  Implementation  Model 

#  Provides  a  framework  for  integrating  often  disparate 
internal  process  management  activities  [e.g.,  quality 
audits,  project  process  status  reporting,  gap  analyses, 
interim  appraisals,  benchmark  assessments]  ^ 
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Near  Term  Plans 


#  ISF  has  been  in  joint  development  between  ISD  Brasil 
and  PUC  (a  Brazilian  University)  since  mid-2006. 


ISF  V0.5  full  scale  pilot(s)  with  several  base  models  and 
maps  Q2/3  2007  (partial  to  full  AW  tool  and  CAM  method 
support) 


#  ISF  VI  .0  initial  release  with  more  base  models  and 
happroveddmaps  Q4  2007/Q1  2008  (full  AW  tool  and 
CAM  method  support) 


i — j 
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Directions,  and  Opportunities 


Issues 

#  Distribution  and/or  importation/integration 
support  for  models  (IP  questions, 
permissions;  not  a  technical  issue) 

#  Definition,  coordination,  acceptance,  and 
maintenance  of  the  model  maps  (more  a 
political  than  technical  issue) 

Status:  ISD  had  obtained  rights  to 
distribute  CMMI  models,  eSCM, 
and  ISO  in  Appraisal  Wizard 

Status:  ISF  itself,  although  ISD 
registered,  is  expected  to  be  in  the 
public  domain. 

Directions 

#  Continue  technical  development  and 
piloting  with  current  interested  parties  (e.g. 
CMU  ITSqc;  global  clients  with  current 

CMMI  and  ISO  requirements;  SSCI) 

#  Continue  to  investigate  and  develop 
solutions  to  legal  and  political  hssuesoin 
collaboration  with  specific  large  influential 
clients,  industry  groups,  and  rstewardsd 

Status:  Engaged  3  global  clients 
already  regarding  pilot  appraisals 
and  development  tasks  (adding 
client  specific  models  of  concern  to 
ISF). 

Status:  Discussing  collaboration  in 
an  SEI  SPRC  (Europe)  initiative. 

Opportunities 

#  Direct  sponsorship  and  collaboration 

#  Collaboration  invitations  from  Consortium 
/  Industry  Association  /  Government 
working  groups 

#  Participation  in  independent  AW  user 
group  with  subcommittees 

#  Creation  and/or  participation  in  a  new 
cross  community  consortium 

Status:  Joined  the  new  Enterprise 
SPICE  initiative  as  part  of  Steering 
Group  and  Development  team. 

Status:  SSCI  sponsored  AW  User 
Group  meeting  scheduled  for 
November  2007.  Follow  on  ISD 
sponsored  AW  User  Group  meeting 
scheduled  immediately  afterwards. 
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Closing  Thoughts 


Process  standardization, 
modeling,  and  improvement 
efforts  are  expanding. 

❖  Process  models  and  frameworks  proliferation  will  continue. 

❖  Independent  model/framework  bodies/owners  are  not  really 
interested  in  giving  up  their  rspace.o 

❖  The  enterprise  cost  impacts  are  significant 

❖  Increased  customer  drivers  for  compliance  is  driving  costs  higher 
when  lower  is  desired. 

Domain  and  business  area 
specific  reference  models  and 
frameworkse 

❖  Directly  address  process  needs  of  specific  sub-communities. 

❖  Do  have  positive  impacts  within  their  constituencies  and  niche 
areas. 

❖  Bute  Can  cause  sub-optimal  investments  in  process,  cause 
counter  productive  implementations,  and  produce  large  expense 
side  inefficiencies 

Mechanisms  being  developed 
and  implemented  by  ISD  accept 
and  address  reference  model 
realities  and  synergies 

❖  ISF®,  appraisal  life  cycle  model,  Appraisal  WizardE  and  Model 
WizardE  V7,  and  CAMSM. 

The  models  can  be  successfully 
integrated  to  improve  enterprise 
performance. 

❖  Improve  both  the  quality  and  efficiency  of  enterprise  process 
improvement  (standardization,  implementation,  management 
oversight,  appraisals) 
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for  Excellence  Benefits 


Operationalize  an  Enterprise 
Improvement  Strategy 

Provides  an  enterprise  strategy  to  implement 
best  practices  from  multiple  models. 

Reduce  compliance  costs 

Leverages  the  commonalities  among  models 
to  reduce  overall  costs  of  compliance. 

Increase  efficiency 

Appraisals  can  be  conducted  using  multiple 
models  simultaneously. 

Provide  a  unified 
implementation  approach 

Provides  management  a  common,  unified 
rroadmapoto  achieve  high  maturity,  high 
performance  goals. 
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A  For  Enterprise  SPICE,  see  the  following  web  site,  under 
finitiatives/Enterprise  SPICE 

V  www.spiceusergroup.org 

A  For  Sarah  Sheard®  current  contact  info: 

T  Principal,  Third  Millennium  Systems  LLC;  sheard@3MilSvs.com 

A  For  ISD  technical  papers  or  AW  download  demo 

V  http://www.isd-inc.com/ 

T  http://members.isd-inc.com/resources.papers/ 

T  http://members.isd-inc.com/support.downloadArea/ 
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Management 
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Results  (performance) 

Leadership,  Alignment  e  Culture 


#  Social  Responsibility  and  Ethics 

#  Leadership 

#  Governance 

#  Strategic  Planning 

#  Strategic  Execution 

#  Culture  of  Excellence 

#  Decision  and  Analysis  Resolution 

#  Audits 
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#  Staffing 

#  Training 

#  Career  Management 

#  Compensation 

#  Performance  Management 

#  Team  Management 

#  Work  Environment 

#  Mentoring  and  Coaching 
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Costumers 
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P&S  Develop. 

P&S  Sustaining 

J 

Management 

l 
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V 
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Results  (performance) 

¥  ”  Costumers 


Costumers 
(post- contract) 


Suppliers  Management 


#  Sourcing  Strategy 

#  Opportunity  Analysis 

#  Sourcing  Planning 

#  Supplier  Selection 

#  Supplier  Management 

#  Quantitative  Supplier  Management 

#  Transition 
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Some  important  definit  ons 


PURPOSE  OF  AN 
ORGANIZATION 


Maximize  shareholders  wealth 


CORPORATIVE 

GOVERNANCE 


Control  mechanisms  and  incentives  so  that  the 
agents  of  a  company  may  have  an  adequate  behavior 
aligned  with  the  shareholders  needs 


INFORMATION 

TECHNOLOGY 

GOVERNANCE 


1)  Decision  rights  and  responsibilities  framework 
which  encourage  IT  adequate  behavior 
2  )  Consists  of  leadership,  structure  and  processes 
to  make  sure  that  IT  will  add  value  to  business 
and  organizations’  strategies  in  a  controlled 
(risks)  and  effective  (ROI)  way 


SUMMARY 


IT  will  help  the  organization  to  maximize 
shareholders  value,  having,  for  such  a 
purpose,  adequate  behavior 
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Market  &  Alignment  &  Human  Infra  &  Measurements  Products  &  Customers 
Competitors  Culture  Capital  Technology  Improvement  Services 
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Excellence-  Maturity  and  Performance 


High  Performance 


Performance 

and 

Culture 


Low  Performance^ 


Level  1 


Initial 


Innovation  and  Improvemeni 
Management  by  Facts 

3d  and  Integrated 


Best  Practices 


Low  Maturity/Capability 


High  Maturity/Capability 
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Frameworks 
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Look,  by  Sarah 
Sheard  of  SPC,  now 
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Ml®-Based  “System  Architecture 


Process  Management 

Organization 
Process  Focus  (OPF) 

Organization  Process 
Definition  (OPD) 

Training  Program  (OT) 

Organization  Process 
Performance  (OPP) 

Organization  Innovation 
and  Deployment  (OID) 


Project  and  Supplier  Management _ 

Project  Planning  (PP)  and  Project  Monitoring  and  Control  (PMC) 
Supplier  Agreement  Management  (SAM) 

Integrated  Project  Management  (IPM)  and  Risk  Management  (RSKM) 
Quantitative  Project  Management  (QPM) 


Pre-Development 

Processes 

0 

Product  Mgt. 

0 

u 

Post-Development 

Processes 

Concept  Exploration 
System  Allocation 
(RD,  TS) 

Proposal  Management 
Project  Initiation  (PP) 

Requirements  Mgt. 
and  Development  (RD) 

Product 

Engineering  (TS,  PI) 

Verification,  Validatior 
(VER,  VAL) 

Tailoring 

Environment 

Conversion 

Installation 

Operation 

Support 

Maintenance 

Retirement 

Integral  Support  Processes 

Quality  Assurance  (PPQA),  Config.  Management  (CM),  Measurement  and 

Analysis  (MA),  Decision  Analysis  and  Resolution  (DAR),  Causal  Analysis 
andResolution  [ CAR ) _ 

49 


®  2005-7,  Integrated  System  Framework  for  Excellence  or  ISF  for  Excellence  -  All  rights  reserved  -  ISD  Brasil 


T^PDF 

■»  Complete 

Your  complimentary 
use  period  has  ended. 
Thank  you  for  using 
PDF  Complete. 

upgrade  to 

yes  and  Expanded  Features 

Aligning  CMMI  &  ITI 

here  Am  1  and  hich  ay  Do  1  Go 

L _ 

_ J 

cognenceinc 

Improving  Software  Economics 

©2006  -  cognence,  inc. 


I  APDF 

■»  Complete 


Your  complimentary 
use  period  has  ended. 
Thank  you  for  using 
PDF  Complete. 


and  Expanded  Features 


•  Where  Am  I? 

-  Current  Situation 

-  Process  Improvement  Objectives 

•  How  Do  I  Get  There? 


-  CMMI 

-  ITIL 

-  Mapping,  Commonalities,  Differences 

•  What  Do  I  Do  Now? 

•  Conclusions 


cognericeinc 

cognence,  inc.  Improving  Software  Economics 


Slide 


cognenceinc 

Improving  Software  Economics 

©2006  -  cognence,  inc. 


Complete 


Your  complimentary 
use  period  has  ended. 
Thank  you  for  using 
PDF  Complete. 


Wwfj- '  ?es  and  Expanded  Features 


on 


Satisfying  market 
pressures: 


Improve  Customer 
Satisfaction 


Increase  quality  of  product 
and  service  delivery  and 
support  while  reducing  costs 


cognericeinc 

cognence,  inc.  Improving  Software  Economics 


Slide 


*  Complete 


Your  complimentary 
use  period  has  ended. 
Thank  you  for  using 
PDF  Complete. 


and  Expanded  Features 


Proliferation  of 
“quality”  ^3 
groups 


acking  communication 
&  coordination  with 
others 


Ignoring  linkage 
to  usiness  goals 


Creating  uni  ue  process 
repositories, 

architectures,  data  stores 


Focus  on  their  own  1  |  T I 
model,  standard,  ISo 
requlatory  mandate  £ 

CM  U  - 


Compete  for  limited 
corporate  resources 
&  udgets 


Training 


Surveys 


Assessment 


Metrics 

Collection 


cognence,  inc. 


cognericeinc 

Improving  Software  Economics 


Slide 


I  APDF 

■»  Complete 


Your  complimentary 
use  period  has  ended. 
Thank  you  for  using 
PDF  Complete. 


and  Expanded  Features 


vement  0  ectives 


•  Standardized  approach  to  process  definition  and 
implementation  in  a  variety  of  environments 
(product  and  service  development) 

-  Alignment  to  business  needs 

-  Proven  best  practices  that  deliver  measurable  results 

-  Organizational  structure  for  development  of  processes 
and  procedures 

-  Common  language  both  internally  and  externally 
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•  A  structured  collection  of  practices  that  describes  the 
characteristics  of  effective  processes 

•  Integration  of  common  elements  and  best  features  of 
multiple  CMMs,  providing 

-  Common  terminology 

-  Training 

-  An  integrated  appraisal  method  (SCAMPI*) 

•  CMMI  enables  a  functional  integration  of  all  disciplines 
required  to  develop  a  product  or  service 


SCAMPISM  Standard  CMMI  Appraisal  Method  for  Process  Improvement 
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Standard, 
Consistent  Process 


Disciplined 


2: 

Managed 


Continuously 
Improving  Process 


Predictable 


Quantitatively 

Managed 


Allofourproects 
follow  similar 
processes,  e 
know  what  to  do 
when. 


hat  Process 


Things  are  getting  under 
control  -  our  schedules  and 
plans  now  include  most  of 
what  we  have  to  do. 


5: 

Optimizing 


t 

e  know 

uantitatively  in 
advance  when 
pro  ects  are  going 
well  or  not,  and 
we  intelligently 
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improve. 
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System  &  Software  Development 


-  Entire  lifecycle  or  part  thereof 

-  All  types  of  lifecycle  methods 

-  Including  sustainment  aspects  of  system  development 

•  Logistics,  training,  etc 

•  System  &  Software  Maintenance 

•  Providing  services 

-  Training 

-  Delivery/transportation 

-  Infrastructure  and  Operations 
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•  The  'library'  has  evolved  to 
it's  current  version,  JTIL 
v3.  The  five  volumes  are: 


-  ITIL  Service  Strategy: 

-  ITIL  Service  Design: 

-  ITIL  Service  Transition: 

-  ITIL  Service  Operation: 

-  ITIL  Continual  Service 
Improvement, 

-  which  can  be  obtained 
from  the  publishers, 
TSO  Books 
(www.tsoshop.co.uk) 
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•  The  ITIL  volumes  make  up  a  comprehensive,  non-proprietary 
and  publicly  accessible  process-related  library  in  the  IT  field. 

It  covers: 

-  Descriptions  and  definitions  of  the  various  ITIL  practices  and 
disciplines 

-  Organizational  structure  and  skill  requirements 

-  Best  practices  and  processes  for  planning,  provision  and  support 
of  IT  services 

•  ITIL  is  a  registered  trademark  of  the  UK  Government's  Office 
of  Government  Commerce  (usually  known  as  the  OGC) 
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Processes 
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and  improving 
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transitioning  new  changed 
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Service 

Transition 


Capa  ilities  for  managing 
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Service 

Strategy 
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Service 

Design 


r 


Service 

Transition 
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Service 

Operations 


ITI  Processes 


•Service  evel  Mgmt. 
•Capacity  Mgmt. 
•Availa  ility  Mgmt. 
•Continuity  Mgmt 
•Service  Catalogue 
Mgmt. 

•Information  Security 
•Supplier  Mgmt. 


•Transition  Planning  & 
Support 
•Change  Mgmt. 

•Asset  &  Config.  Mgmt 

•  Release  &  Deployment 
Mgmt. 

•  alidation  &  Testing 
•Evaluation 

•  nowledge  Mgmt. 


•Incident  Mgmt. 

•Pro  lem  Mgmt. 

•Event  Mgmt. 

•Re  uest  Fulfillment 
•Access  Mgmt. 
•Operational  Activities 
Covered  in  other  Phases 


•  Step  Improvement 
Process 

•Service  Reporting 
•Service  Measurement 

•  ROI  for  CSI 

•Business  Questions  for  CSI 
•Service  evel  Mgmt. 


•Blue  denotes  new  in  ITI  v 
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pping 


Service 

Strategy 


a 


Service 

Design 


r 


Service 

Transition 


k 


V 


Service 

Operations 


CM  Ml  Process  Areas 


•  Re  uirements  Mgmt. 

•  Re  uirements  Dev. 
♦Proect  Planning, 

Pro  ect  Monitoring  & 
Control 

•  Integrated  Pro .  Mgmt. 
’Quant.  Pro .  Mgmt 
•Meas.  &  Analysis 
•Org.  Process  Focus 
•Org.  Process 

Definition 


•Re  uirements  Mgmt. 

•  Re  uirements  Dev. 
•Technical  Solution 

•  alidation 
•Decision  Analysis  & 

Resolution 

•Integrated  Pro .  Mgmt. 
•Supplier  Agree.  Mgmt 
•Configuration  Mgmt. 


•Configuration  Mgmt 
•Product  Integration 

•  erification 

•  alidation 

•  Product  &  Process 
Quality  Assurance 


•Proect  Monitoring  & 
Control 
•  alidation 
•Causal  Analysis  & 
Resolution 


•Measurement  &  Analysis 
•Product  &  Process  Quality 
Assurance 
•Org.  Process 
Focus  Definition 
•Org.  Process  Performance 
•Quantitative  Pro  ect  Mgmt. 
•Causal  Analysis  & 
Resolution 

•Org.  Innovation  &  Deploy. 
•Integrated  Pro  ect  Mgmt. 
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CMMI  vs.  ITI 
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Em  odies  years  of  product  or  service  development 

Contains  tried,  tested  and  enhanced  practices 

Esta  lishes  process  &  process  improvement  policy 

Deals  with  roles  and  responsi  ilities 

Ensures  training  and  skills  of  resources 

Provides  guidance  to  measure  performance 

elps  to  set  process  improvement  o  ectives  &  priorities 

elps  to  ensure  sta  le,  capa  le,  and  mature  processes 

Guides  improving  pro  ect  &  organi  ational  processes 

ith  an  appraisal  audit  method  to  determine 

maturity  compliance  for  registration  -  oth  indicators  of 

a  ility  to  deliver  products  and  services 
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CMMI 


vs.  ITI 
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/ 
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Strategi  e  a  usiness  service  definition  and  catalog  to  drive 
process  design,  transition,  operations  and  improvement 

Esta  lish  and  define  process  improvement  infrastructure 

Provide  descriptive  tools,  technologies,  techni  ues  &  standards 

Defined  roles  and  responsi  ilities  with  skill  descriptions  and 
training  re  uirements 

Plan  and  develop  products  services 

Integrate,  deploy  and  transition  product  services 

Measure  and  improve  product  service  delivery 

Benchmark  via  maturity  or  capa  ility  level 

Provide  industry  recognition  through  registration 
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CMMI  Process  Areas 
(Project  Management) 

Related  Business  0  ectives  of  Services 
Organi  ations 

ITI  ife  Cycle  Phase 

Project  Planning 

Maintain  detailed  service  plans  that  include  the  budget  and 
schedule  needed  to  support  the  customer 

Service  Strategy 

Project  Monitoring  &  Control 

Manage  the  costs  and  schedule  associated  with  the 
service 

Service  Strategy 

Supplier  Agreement 

Management 

Effectively  manage  suppliers  of  tools  or  resources  vital  to 
the  success  of  the  service 

Service  Design 

Integrated  Project  Management 

Integrate  the  delivery  of  the  service  with  other  projects  and 
stakeholders 

Service  Strategy;  Service 

Design 

Risk  Management 

Plan  for  current  and  future  risks  to  the  service  project 

All 

Quantitative  Project 

Management 

Establish  predictability  in  their  services 

Service  Strategy;  CSI 

Adapted  from  CM  SEI  T 
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CMMI  Process  Areas 
(Support) 

Related  Business  0  ectives  of  Services 
Organi  ations 

ITI  ife  Cycle  Phase 

Configuration  Management 

Control  technical  and  management  work  products 

Service  Transition 

Process  &  Product  Quality 
Assurance 

Ensure  their  services  meet  quality  objectives  and  customer 
requirements 

CSI;  Service  Transition 

Measurement  and  Analysis 

Understand  measures  of  cost,  profitability,  &  cost  of  quality 

Service  Strategy;  CSI 

Decision  Analysis  and 

Resolution 

Make  informed  and  justifiable  selections  of  products  or 
techniques  for  their  customers 

Service  Design 

Causal  Analysis  and  Resolution 

Track  service  issues  to  root  causes  and  eliminate  them 

Service  Operation;  CSI 

Adapted  from  CM  SEI  T 
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CMMI  Process  Areas 
(Engineering) 

Related  Business  0  ectives  of  Services 
Organi  ations 

ITI  ife  Cycle 
Phase 

Requirements  Management 
Requirements  Development 

Develop  and  manage  their  service  requirements 

Service  Strategy;  Service 
Design;  (Service. 
Operations) 

Technical  Solution 

Provide  services  that  provide  technical  stability  and  support 
all  aspects  of  product  development  and  fielding 

Service  Design;  (Service 
Operations) 

Product  Integration 

Ensure  that  interfaces  are  compatible  prior  to  their 
integration 

Service  Transition; 

Service  Operation 

Verification 

Confirm  that  performed  services  satisfy  their  service 
requirements 

Service  Transition; 

(Service  Operation) 

Validation 

Evaluate  the  suitability  of  acquired  products  &  services 

Service  Design;  (Service 
Transition  &  Operation) 

Adapted  from  CM  SEI  T 
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CMMI  Process  Areas 
(Process  Management) 

Related  Business  0  ectives  of  Services 
Organi  ations 

ITI  ife  Cycle 
Phase 

Organizational  Process  Focus 

Build  and  maintain  a  service  culture 

Service  Strategy;  CSI 

Organizational  Process  Definition 

Implement  and  improve  processes  to  support  predictable 
successful  execution 

Service  Strategy;  Service 
Design;  CSI 

Organizational  Training 

Train  staff  members  to  perform  service  functions 

All 

Organizational  Process 

Performance 

Measure  effectiveness  &  performance  of  processes 

Service  Strategy;  CSI 

Organizational  Innovation  & 
Deployment 

Introduce  new  service  methods,  technologies,  &  functions 

Service  Strategy;  CSI 

Adapted  from  CM  SEI  T 
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d  ects”  at  Several  evels 


To  use  CMMI  and  ITIL  together, 

•  Define  an  “Project”  at  three  levels 

1 .  Creation  of  Service  Offering  -  treat  your  service 
catalogue  and  service  portfolio  as  its  own  project  (ITIL  & 
CMMI) 

2.  Definition  of  a  development  project  (CMMI) 

3.  Manage  service  operations  -  identifying  the  service 
catalogue  item(s)  as  project 


•  ITIL  to  define 

•  CMMI  to  appraise 
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;ture  for  Implementing  “Pro  ects” 


Minimum  Delivera  les  Ta  le 


Lifecycle  Phases  and  Associated 
Core  Processes 


Initiating  Phase 

•  Change  Management  Rocedure 

•  Roject  Panning  Rocedure 

•  Roduct  Engineering  Rocedure 

•  Risk  Management  Rocedure 

•  Measurement  &  Analysis  Rocedure 


Large  Projects >  1000  hours  -  New 
Engagement  |  Service  Catalog 
Entry 

Y  Hich  level  estimate  (with  rationale) 

Y  Hich  level  Requirement  sQesigi 

Y  Rsk  log,  Meeting  minutes 

Y  Roject  contrai  (eg.,  SOW,  SOS) 

Y  Fteer  review  (estimate,  rqts,  designs,  risks) 

Y  CQ  or  Change  ticket 


W  ork  Products 
Medium  Projects 
(200  -  999  hours,  App 
Enhancement,  Incident  1 1  M/AM 
Combined  |  Contract  Amendment 

Y  High  level  estimate  (with  rationale) 

Y  High  level  Requirements/Desigi 

Y  Rsk  log,  Meeting  minutes 

Y  Roject  contraS  (SOW,  SOS  pool  hours) 

Y  Fteer  review  (estimate,  reqts,  design,  risks) 

Y  CQ  or  Change  ticket 


Small  Projects 
(<  200  hours),  Service 
Requests  |  IM  Projects  | 
IM/AM  Combined 

Y  CQ  ticket  or  Change  ticket 

Y  Meeting  minutes 


Planning  Phase 

•  Roject  Panning  Rocedure 

•  Integrated  Teaming  Rocedure 

•  Requirements  Mgnt  Rocedure 

•  Rsk  Management  Rocedure 

•  Configuration  Mgnt  Rocedure 

•  Knowledge  Management  Rocedure 

•  Supplier  Management  Rocedure 

•  Environment  Mgnt  Rocedure 

•  Roject  Monitoring  Rocedure 

•  Quality  Assurance  Rocedure 

•  Measurement  &  Analysis  Rocedure 

•  Change  Management  Rocedure 


Y  Roject  Mgmt  Han  (PMP)  (with  project 
glossary,  lifecycle,  tools  project  schedule, 
WB§  risk  plans) 

Y  Roject  Monitoring  and  Control  Ran  (could 
be  incorporated  into  the  PGP) 

Y  Configuration  Management  Ran 

Y  Detailed  project  estimate  (effort,  schedule, 
cost,  size) 


Y  Roject  Mgnt  Ran  (PMP)  (with  project 
cjossary,  lifecycle,  tools  project  schedule, 
WB$  risk  plans) 

Y  Roject  Monitoring  Ran  and  Control  Ran 
(could  be  incorporated  into  the  FGF) 

Y  Configuration  Management  Ran 

Y  Detailed  project  estimate  (effort,  schedule, 
cost,  size) 


Y  Roject  Mgmt  Ran  (PMP)  (e.g., 
annual  plan,  service  line) 

Y  Configuration  Management  Ran 
(e.g.,  annual  plan,  service  line) 

Y  Fteer  review  records  for  PGP, 
Configuration  Management  Ran, 
and  estimate 

Y  CQ  ticket  or  Change  ticket 


Y  Fteer  review  records  for  PGP, 
Configuration  Management  Ran,  and 
Detailed  estimate 


Y  Fteer  review  records  for  PGP,  Y  Meeting  minutes 

Configuration  Management  Ran,  and 
Detailed  estimate 


Y  Roject  budget 

Y  Meeting  minutes,  Issues  Log  Rsk  Log 


Y  Roject  budget 

Y  Meeting  minutes,  Issues  Log,  Rsk  Log 
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•  Think  of  commonalities  of  CMMI  and  ITIL  instead  of  the 
differences  -  great  synergy 

•  Think  of  the  CMMI  as  the  “what”  and  ITIL  as  the  “how” 

ITIL  -  where  it  provides  detailed  processes,  techniques, 
templates  and  tools 

•  Keep  “Maturity”  as  the  goal  (or  Capability) 

Utilize  CMMI  process  improvement  infrastructure  (e.g., 
Executive  Steering  Committee,  Process  Engineering  Group, 
etc.)  to  facilitate  continual  service  improvement  -  as  defined 
in  CSI  volume 

Use  CMMI  to  assess  what  is  being  done 
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mil  Mages 

•  ITIL  Online;  the  official  ITIL  Page  http://www.itil.co.uk 

•  OGC  Office  of  Government  Commerce  http://www.oqc.qov.uk 

•  The  ITIL  and  ITSM  Directory  http://www.itil-itsm-world.com/ 

•  IT  SMF  Forum  http://www.itsmf.com/ 

•  ITIL  Portal  http://www.interpromusa.com/ 

•  Service  Management  Institute  http://www.itsmi.com 

•  EXIN  ITIL  Examination  Institute  http://www.exin-exams.com/ 

•  Where  you  can  get  the  ITIL  books  http://www.itilbooks.com/ 

•  ITIL  Community  www.itilcommunitv.com 
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Pat  Mitryk 
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732.575.5445 
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Background 


NORTHROP  GRUMMAN 


■  Predictive  modeling  is  an  essential  skill  at  CMMI  Levels  4  and  5 

-  Organizational  Process  Performance  requires  predictions  based 
on  statistical  analysis  of  the  organization’s  standard  process 

-  Quantitative  Project  Management  requires  predictions  based  on 
statistical  analysis  of  the  project’s  defined  process 

■  Predictive  modeling  relies  on  historical  program  performance 
data  (predictive  analytics)  in  conjunction  with  a  forecasting 
algorithm  model  to  predict  future  outcomes 

-  Ranges  from  simple  extrapolation  techniques  to  sophisticated 
Neural  Network  based  models 

■  This  presentation  will  discuss  the  principles  of  predictive 
modeling,  outline  the  fundamental  methods  and  tools,  and 
present  typical  results  from  applying  these  techniques  to  project 
performance 
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s  What  is  Predictive  Analysis? 

■  Recent  Trends 

■  Application  to  Program  Performance 

■  Summary 
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What  is  Predictive  Analysis? 
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■  Could  this  network  packet  be  from  a  virus  attack? 

-  Predict  likelihood  of  the  network  packet  pattern 

Anomaly  detection  (outlier  detection) 

-  Similar  questions: 

•  Are  the  hospital  lab  results  normal  (Adverse  drug  effect  detection) 

•  Is  this  credit  transaction  fraudulent?  (fraud  detection) 

■  Will  this  student  go  to  college? 

-  Based  on  Gender,  Parentlncome,  ParentEncouragement,  IQ,  etc. 

-  E.g.,  if  ParentEncouragement=Yes  and  IQ>1 00,  College=Yes 
4  Classification  (prediction) 

-  Similar  questions: 

•  Is  this  a  spam  email?  (spam  filtering) 

•  Recognition  of  hand-written  letters  (pen  recognition) 

■  What  is  the  person’s  age? 

-  Based  on  Hobby,  MaritalStatus,  NumberOfChildren,  Income,  HouseOwnership, 
NumberOfCars,  ... 

-  E.g.,  If  MaritalStatus=Yes,  Age  =  20+4*NumberOfChildren+0.0001*lncome+... 

Regression  (prediction) 
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■  What  is  Predictive  Analysis? 
s  Recent  Trends 

■  Application  to  Program  Performance 

■  Summary 
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Predictive  Analysis  Trends  -  Adoption  is  on  the  rise 


■  Predictive  Analysis  is 
becoming  more  prevalent 
and  integrated  in  business 
applications 

o  Example:  Disease 
management  and  evidence 
based  care,  based  on 
historical  diagnosis  and 
procedure  codes  of  patients 

o  Example:  E-Mail  filtering 
using  predictive  analysis 

■  Predictive  Analysis 
algorithms  are  being 
integrated  into  existing 
databases,  data  mining 
tools 

o  Example:  Microsoft  SQL 
Server  2005  has  predictive 
analysis  algorithms 


Example: 

Premium  predictive  analysis  based  filtering  on  e- 
mail,  available  to  any  e-mail  user 
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Predictive  Analysis  Trends  -  Tools  are  becoming  easier  to  use 
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Data  Mining  / 
Predictive  Modeling 


Online  Analytical 

Processing 


Dashboards 


Reports  (Adhoc) 


Reports  (Static) 


Easy 


Usability 


Difficult 
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Predictive  Analysis  Trends  -  Model  development  is  more  structured 


Training  Data 


Train  the  Model 


Test  Data 

Test  the  Model 

Prediction  using 
the  Model 


Prediction  Input  Data 


Third  Party 

Off-the-shelf  or  \  Predictive 
Proprietary  Predictive)  Analysis 
Analysis  Engine  /  tools 


Off-the-shelf  or  proprietary 
Predictive  Analysis  Model 


■  Executive  understanding  of  the  creation,  training 

and  testing  of  the  model  is  critical  to  success 

■  The  Model  gets  more  powerful  and  accurate  as 

rnn 

ine  volume  ot  aaia  Tea  imo  ine  moaei  increases  - 
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Predictive  Analysis  Trends  -  Algorithms  are  available  for  use 
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Classification 

Regression 

Segmentation 
Association  Analysis 

Anomaly  Detect. 

Sequential  Analysis 

Time  series 


1  -  First  Choice 


2  -  Second  Choice 
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Data  Mining  Vendors  &  Tools 


■  SAS  (Enterprise  Miner) 

■  IBM  (DB2  Intelligent  Miner) 

■  Oracle  (ODM  option  to  Oracle  lOg) 

■  SPSS  (Clementine) 

■  Insightful  (Insightful  Miner) 

■  KXEN  (Analytic  Framework) 

■  Prudsys  (Discoverer  and  its  family) 

■  Microsoft  (SQL  Server  2005) 

■  Angoss  (KnowledgeServer  and  its  family) 

■  DBMiner  (DBMiner) 

■  Many  others 
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■  What  is  Predictive  Analysis? 

■  Recent  Trends 

v'  Application  to  Program  Performance 

■  Summary 
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Mission  Assurance  Continuum 
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Predictive 
Program  Health 


Industry  Innovators 


Proactive  Program  Management 
Program  Portfolio  Management 

Approach  and  Scope 

•  Self  reported  Program  Portfolio  includes  critical 
and  high  visibility  programs 

•  Standard  Program  Management  Metrics 
collected  on  a  periodic  basis 

Infrastructure  and  Breadth 

•  Program  data  maintained  by  individual  programs 

•  Summary  information  provided  to  enterprise 
repository 


Data  Requirements 

•  Very  few  metrics  collected  from  programs 

•  Key  program  metrics  (cost  performance, 
schedule  performance,  technical  performance, 
CPI,  SPI  etc.) 

•  Standardized  program  taxonomy  information  like 
customer,  contract  type  etc. 


Reports  based  on  current  and  passed 
performance  data  of  portfolio  programs, 
programs,  and  subcontract  reports 


Self  Reported  Program  metrics  collected  periodically 
and  at  specific  program  milestones 
Reporting  analysis  performed  as  needed 


Program  data  collected  periodically  into  an  enterprise¬ 
wide  program  management  repository 
Program,  Enterprise  and  Subcontracts  performance 
reports  available 


25-  100  metrics  collected  from  programs 

Key  program  metrics  collected  at  all  specified  Program 

Milestones. 


Predictive  Analysis  based  on 
Program  Performance  Modeling 

Self  reported  program  metrics,  organizational  data, 
personnel  data  and  customer  reported  metrics 
collected  at  regular  intervals 
Predictive  models  developed  using  historical  data 
(leading  indicators  rationalized) 

Models  validated  against  historical  data 

Holistic  enterprise  wide  approach  to  program  execution 
Models  continually  refined  using  current  program 
performance  data 

Sophisticated  predictive  measures  provided  to  programs 
and  enterprise 

50  -  75  metrics  collected  from  programs  and  refined  to 
include  only  the  few  relevant  metrics 
Adaptive  approach  to  qualitative  and  quantitative 
performance  indicators 

Direct  and  Indirect  metrics  collected  for  the  programs; 
qualitative  information  is  mined 
Proactive  responses  based  on  predictive  analysis  of 
ongoing  and  historical  performance 
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Overarching  Objectives  for  Predictive  Modeling 
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■  Provide  program  management  staff  with  Predictive  Models  to 
“test-their-gut”  against  enterprise  experience  data  before 
making  strategic  program  decisions 

■  Develop  Predictive  Models  that  provide  insight  into 
identifying  “headlight  metrics”  that  influence  Schedule  and 
Cost  realism  during  program  execution 

■  Leverage  existing  enterprise  information  to  develop 
Predictive  Models  for  programs 

■  Ensure  that  models  are  extensible  and  automatically 
calibrated  with  additional  data  from  the  program  and 
enterprise 
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Potential  Areas  for  Predictive  Analysis 


NORTHROP  GRUMMAN 


Potential  Predictive 
Analysis  Models  for 
Program  Management 
and  Subcontractor 
Management 

•  Schedule  Risk  at  WBS  level 
based  on  past  performance 

•  Cost  Risk  at  WBS  level 
based  on  past  performance 

•  Technical  Risk  at  WBS  level 
based  on  past  performance 

•  Spending  and  staffing  profile 
for  the  program  life  cycle 

•  Subcontractor  risk  profile 
based  on  past  performance 

•  Sub-tier  quality  at 
subcontract  and  WBS  level 

•  Defect/Aberrations  for  the 
program  life  cycle 

•  Mission  Assurance  models 
based  on  program  category 


Predictive 

Analysis 

Algorithms 

•  Decision  Trees 

•  Naive  Bayesian 

•  Clustering 

•  Sequence 
Clustering 

•  Association 
Rules 

•  Neural  Network 

•  Time  Series 

•  Custom  Model 
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Predictive  Analysis  High  Level  CONOPS 


1)  Enterprise  data  is  mined 
and  analyzed 

2)  Enterprise  models  are 
defined  by  Analysts 

3)  Enterprise  model 
outputs  are  defined  by 
Analysts  and 
customized  by  PM  staff 

4)  PM  staff  use  models 
interactively 

Key  Benefit: 

Leverages  enterprise 
experience  data  and 
sophisticated  algorithms 
into  predictive  models  for 
cost  and  schedule  realism 
checks  during  program 
execution 


V 


Predictive  Analysis 


Predictive  Analysis 
Output  Lift  Graph 


Predictive  Analysis 
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The  Predictive  Modeling  Process 


■  Explore  the  Data 

■  Understand  Data 
Relationships 

■  Derive/Enhance  the 
Data 

"  Use  the  Data  to 
Predict 

■  Train  the  Model 


Train 


r 


Collect  /  Derive 


Test 


Copyright  ©  2007  Unlimited  Innovations,  Inc. 


1 R 


NORTHROP  GRUMMAN 


Rationalization 


Preparation 


Analysis 


Qualify  Associations 


Research  Strength 
of  Relationships 


Association  /  Clustering 


Determine  Prediction 
—  Quality 


What  Can  Be 
Predicted? 


— 


Make  a  Prediction 


Neural  Network 
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What  can  be  Predicted  with  Reasonable  Accuracy? 
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Large 
volume  of 
historical 
data 


<D 

o 

*< 

o 

(D 


(/> 

fi) 

(D 

(D 


Limited 

Historical 

data 


Limited  Number  of 

Programs 

Enterprise 

Experience 

■  Likelihood  or  return  to 

■  Quadrant  2  predictions 

acceptable  performance 

■  Quadrant  3  predictions 

■  Predictive  Program 

■  Early  warning 

Performance 

“headlight  indicators” 

■  Higher  accuracy  based 

on  enterprise 

1 

experience  3 

■  Cost,  schedule  realism 

■  Phase  realism 

■  WBS  Accuracy 

2 

Low  High 


Volume  of  “Like”  Programs 
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Derivation  of  Data  &  Data  Relationships 
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■  Examples  of  Derived  Data 

-  Number  of  Outstanding  Program  Issues  (with  and  without  recovery  dates) 

-  Variance  in  program  Cost/Schedule/Technical  health  from  month-to-month 

-  Program  Cost/Schedule/Technical  health  trend  from  month-to-month 

-  Variance  in  VAC  from  month-to-month  taken  as  a  percentage  of  the 
current  EAC 

■  Examples  of  Discovered  Relationships 

-  Schedule  Health  is  a  good  indicator  of  program  Overall  Health  recovery 

-  Cost  and  Technical  Health  are  good  indicators  of  program  Overall  Health 
decline 


Better  understanding  of  the  data  allows  for  organization  and 

enhancement  of  the  dataset 


Copyright  ©  2007  Unlimited  Innovations,  Inc. 


1ft 


Empowering  Your  Enterprise 


Unlimited - — -  NORTHROP  GRUMMAN 

Innovations  - - 


Model  Development  &  Calibration 

Model 


•  Modeling  without  applied  domain  knowledge 
or  calibration  resulted  in  lower  accuracy 

•  Association  models  able  to  determine 
relevant  data  attributes 


Calibrated  Model 


•  Incorporating  domain  knowledge  and 
calibration  into  data  mining  resulted  in  higher 
accuracy 

•  Data  relationships  are  more  clearly  defined 


Domain  knowledge  &  calibration  applied  to  data  mining  can 

enhance  the  predictive  model 
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Presentation  of  the  Results  from  the  Models 


Ability  for  Programs  to  review  the 
predictive  output  from  multiple 
models  to  “test-the-gut”  before 
making  strategic  program  decisions 
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Presentation  of  the  Results  from  the  Models 


Ability  for  staff  to  review  status  and 
trends  across  the  portfolio  of 
programs,  across  a  variety  of 
categories 
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Algorithm  References 
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■  Decision  trees  (classification/regression): 

-  ftp://ftp.research.microsoft.com/users/suraiitc/icde99.pdf 

-  http://www.research.microsoft.com/research/pubs/view.aspx7tr  id=81 

-  http://research.microsoft.com/~dmax/publications/dmart-final.pdf 

■  Association  rules: 

-  Apriori  algorithm  (see  Data  Mining  concepts  and  techniques) 

■  Clustering 

-  EM:http://www.research.microsoft.com/scripts/pubs/view.asp?TR  ID=M 

SR-TR-98-35 

-  K-means  (see  Data  Mining  concepts  and  techniques) 

■  Sequence  clustering 

-  ftp://ftp.research.microsoft.com/pub/tr/tr-20QQ-18.pdf 

■  Time  series: 

-  http://research.microsoft.com/~dmax/publications/dmart-final.pdf 

■  Neural  network 

-  Conjugate  gradient  method  (see  Data  Mining  concepts  and  techniques) 

■  Naive  Bayesian 

-  See  Data  Mining  concepts  and  techniques 
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More  Information 
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■  OLE  DB  for  DM  specification 

-  http://www.microsoft.com/downloads/details.aspx?FamilvlD=01005f92- 

dbal  -4fa4-8ba0-af6a1 9d3021 7&DisplavLang=en 

■  Plug-in 

-  http://www.msnusers.com/AnalvsisServicesDataMininq/Documents/File 

s%2FSQL%20Server%20Data%20Mininq%20Pluq%2Dln%20Alqorithm 

s%20%28Beta%202%20%2B%2B%29.zip 

-  A  white  paper,  tutorial,  and  complete  sample  code  for  Pair-wise  Linear 

Regression 

■  SQL  Server  2005: 

-  www.microsoft.com/sql/2005 

■  Community: 

-  Microsoft.public.sqlserver.datamining 

-  Microsoft.private.sqlserver2005.analvsisservices.datamininq 

-  Groups.msn.com/AnalvsisServicesDataMininq 

■  msdn.microsoft.com  (search  “data  mining”) 
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Agenda 
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■  What  is  Predictive  Analysis? 

■  Recent  Trends 

■  Application  to  Program  Performance 
■S  Summary 
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Summary  -  Critical  success  factors 
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■  Executive  and  Enterprise  support  and 
understanding  of  long-term  strategic  benefits 

■  Understanding  of  the  types  of  data  and  the 
correlation  between  the  data 

■  Understanding  of  the  various  constituents  in  the 
value  chain  and  the  tools/processes  for  each 
constituent 

■  Prototypes  or  mockups  that  depict  the  results  of 
the  model 

■  Sound  and  robust  technical  architecture 

■  Delivery  mechanism  that  shields  the  complexity  of 
the  model  from  the  end  users 


OR 
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Contact  Information 


Philip  Paul 

Unlimited  Innovations,  Inc. 
(714)  342-8952 
ppaul@uius.com 
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(310)  812-7290 
rick.hefner@nqc.com 
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•  Mature  CMMI®  organizations,  teamed  together,  do 
not  automatically  yield  a  mature  program  team 

■  ML3  org  +  ML3  org  +  ML3  org  \  ML3  team 

•  Immature  organizations  tend  to  drive  down  mature 
organizations 

■  Immature  organizations  may  not  have  the  qualifications 
to  participate  in  mature  organizational  behaviors 

■  Unless  all  team  members  participate,  there  may  be 
nattritiono 


The  rteam®  maturitydwill  most  likely  start  out  at  level  1 ! 


®CMMI  is  registered  in  the  U.S.  Patent  and  Trademark  Office  by  Carnegie  Mellon  University. 
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Management  Skill 


Mismatch 

Mature  Team  Member 
expects  high  maturity 
behavior  from  Prime 


Outcome  not 
predictable — Behavior 
of  Prime  will  likely 
dominate 


Disaster 

No  discipline 
No  process 
No  product  (or  low 
Quality  product) 


Low 


Matched 

Team  Members  are  both 
high  maturity 

Highest  probability  of 
success  BUT  defined 
team  processes  are 
needed 


Mature  Prime  may  expect 
lower  maturity  Team  Member 
to  behave  maturely 

Lower  maturity  team  member 
may  not  be  capable  of  higher 
maturity  behavior 


Team  Member  A  (Prime) 
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•  A  mature  program  team  needs  a  defined 
program  process  but  the  source  isndi  obvious 

■  Each  team  member  will  likely  have  its  own 
Organizational  Standard  Process 

■  Dictating  using  the  Prime®  process  is  usually  not 
feasible  and  introduces  unnecessary  risk 

■  Formulating  a  brand  new  process,  never  deployed 
by  any  team  member,  is  also  risky 


What®  the  best  choice  for  a  multi-organizational  team®  process? 


®CMMI  is  registered  in  the  U.S.  Patent  and  Trademark  Office  by  Carnegie  Mellon  University. 


November  13,  2007 


Copyright  2007  Lockheed  Martin  Corporation 


5 


I  APDF 

■»  Complete 


Your  complimentary 
use  period  has  ended. 
Thank  you  for  using 
PDF  Complete. 


and  Expanded  Features 


lem  Statement  -  3 


•  The  problem  is  exacerbated  if  team  members 
are  at  different  CMMI  maturity  levels 

■  Behavior  differs,  depending  on  the  maturity  level 

■  Team  members  at  lower  maturity  levels  may  not 
understand,  appreciate  or  expect  higher  maturity 
behaviors 

■  Failure  to  integrate  subcontractors  can  result  in 
lack  of  commitment  for  end  item  responsibility, 
depending  on  subcontractor®  role/integration 

■  Stakeholders  with  varying  maturity  have  different 
expectations 


The  problem  can  be  amplified  if  customers  are  on  the  team. 

®CMMI  is  registered  in  the  U.S.  Patent  and  Trademark  Office  by  Carnegie  Mellon  University. 
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The  Need 


•  Well-defined  program  processes  are  critical  to 
a  program  team®  success 

•  Customers  increasingly  expect  team 
processes  to  be  common,  integrated  and 
mature 

•  A  well-defined  approach  for  program  process 
definition  is  needed  when  forming  teams  of 
individually  mature  organizations 

•  Common  processes  minimize  risk  and 
promote  a  quality  product/service 

•  A  mature  program  approach  to  process 
enables  fproactiveomanagement 
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•  Program  process  definition  based  on 

■  Shared  objectives 

■  Shared  process  needs 

■  Shared  vision 

■  Clearly  defined  roles  and  responsibilities 

•  Common  process  infrastructure 

•  Clearly  defined  interfaces  between  common 
and  unique  processes 

•  Program  process  measurement  in  areas 
critical  to  program  success 
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■7 


•  Program  needs  and  objectives 

•  Program  risks  and  opportunities 

•  Customer  requirements  (RFP,  award  fee,  schedule/cost,  etc.) 

■  Could  include  critical  processes  or  Process  Areas 

■  CMMI  compliance 

■  Other  defined  standards  compliance 

•  Program  management  needs 

■  Program  reviews 

■  Program  reporting  (cost,  schedule,  etc.) 

■  Measurement  (performance,  productivity,  phase  specific,  etc.) 

•  Suppliers  and  Subcontractors 

■  Integrated  Team 

■  End  Item  responsibilities 

•  Work  environment 

■  Co-location  versus  virtual 


The  Challenge:  Translating  drivers  into  the  “right”  Integrated  Team  Process 
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s  the  “Right”  Process? 


The  nrightoprocess  is  one  that 

•  Meets  requirements,  including  standards 

■  From  the  customer 

■  From  the  team  membersoorganizations 

•  Is  appropriately  suited  to  the  domain  and  program 

•  Contains  necessary  and  sufficient  process  elements 

•  Is  integrated  across  the  disciplines  and  team  members 

•  Is  tailored  from  one  of  rtheoorganizational  standard 
processes  of  mature  team  members  or  defined  by  the 
program  team 

•  Is  measurable 

•  Supports  development  of  a  quality  work  product 

Blue  additions  are  new  requirements  for  multi-organizational  team  processes 
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Process  Definition  Approac 


Organization’s  Set 
of  Standard  Processes 


Life-Cycle  Model 
Descriptions 


Process 

Architectures 


Organizational  Assets 


Tailoring 
x^Guidelines 


Organization’s 

Measurement 

Repository 


Organization’s 
Process 
Asset  Library 


Project  Environment 


(  Project  A’s  N, 

(  Project  B’s 

(  Project  C’s 

V  Defined  Process  J 

\Defined  Process  J 

\Defined  Process  J 

Project  A’s 

Project  B’s 

Project  C’s 

Project  Plan 

Project  Plan 

Project  Plan 
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lizational  Team  Process  Definition 
Approach  —  Example 
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Integrated  Program  Team  Process  Architecture 


Team  Member  A  Team  Member  B 

Organizational  Assets  Organizational  Assets 
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OPS 


ram  Process  Infrastructure 

Program’s 
Defined  ¥  Execute. 

i 


Tailored 


Program 

Plan 


Metrics 

Program 

Goals 


Metrics 

Data 


Process 


/ 


Reviews 


Program 
Quant 
Mgt  Plan 

Goals 


Process 

Performance 

Baseline 


Meafc 

Guide 


I 

Product  Quality 


Control  Chart 


Control  Chart 
Program 
Process 
Performance 


\  Pilot 
Improvements 


Causal  Analysis 
&  Resolution 
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Lockheed  Martin 


r 


jrprise  Process  (LM-IEP)  Architecture 


A  comprehensive  enterprise  architecture  serves  as  the  foundation  for  an 

Integrated  Team  Process  Architecture 


Enterprise  Processes  Infrastructure  Processes 


A.1  Organizational  Management 
A.2  Strategic  Planning 
A.3  Quality  Management 
A.4  Ethics  &  Business  Conduct 
A.5  Legal 

A.6  Communications 


B.1  Process  Management 
B.2  Work  Environment  Management 
B.3  Technology  Management 
B.4  Contracts 

B.5  Workforce  Management 
B.6  Finance 

B.7  Supplier  Agreements  &  Procurement 
B.8  Security 

B.9  Property  Management 


Product  Lifecycle  Processes 


*\ 


►  D.3.1  Stakeholders  Needs  Analysis 
D.3.2  Requirements  Development 
D.3.3  Architectural  Design 
D.3.4  Detailed  Design 
D.3.5  Implementation 
D.3.6  Integration 
D.3.7  Verification 
D.3.8  Validation 


Business  Execution  Processes 


Program  Execution  Processes 


Common  Management  Processes 

C.2  C.3  C.4 

Decision  Configuration  &  Performance 

Analysis  Data  Assessment  & 

Management  Control 
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Uninteq  rated 
Process  Elements 


Team  Member 

Influences 


“My  process  is  best” 

That  process  was 
“not  invented  here” 

Not  trained  in 
new/different  process 

“Use  my  process 
architecture” 

“We’re  different” 


The  Challenge:  Seamless  process  interfaces  for  the  Integrated  Team  Process 
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} 


m  Process  Measurement 
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Measurement 


Process 


Requirements  Flowdown  from  the  OSP  to  the 
Project®  Defined  Process 

Planning 


Project  Defined  Process 
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for  Integrated  Program  Team 
cess  £IPTP^I)efinition  - 1 


•  Understand  individual  team  member  process 
capability  (e.g.,  based  on  CMMI  appraisals) 

•  Identify,  for  possible  leveraging,  any  prior 
instances  of  the  same  team  members  working 
on  multi-organizational  program  teams 

•  Define  program  process  needs,  objectives 
and  drivers 

•  Identify  candidate  process  architectures  for 
use  (from  team  members) 
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cess  (IPTP)  Definition  -  2 

•  Convene  process  group  leaders  from  each  team  member 
to 

■  Formulate  the  IPTP  architecture 

■  Identify  needs  for  common  and  unique  processes 

■  Identify  source(s)  for  common  processes 

■  Define  interfaces  across  common  and  unique  processes 

■  Define  process  infrastructure  needs 

•  Establish  mechanism  for  collective  objective  evidence  for 
appraisals 

•  Use  IDEALSM  model  and  common  infrastructure  to 
improve  the  IPTP 

•  If  the  team  is  not  co-located  then 

■  Establish  a  focused  program  process  training  program 

■  Establish  a  process  quality  audit  program  that  visits  each  site 

■  Assign  a  responsible  engineer  at  each  site  to  monitor  process 

SM  IDEAL  is  a  service  mark  of  Carnegie  Mellon  University. 
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he  IDEAL  Model 


Infrastructure  charter  is 
key  to  Initiating  Phase 


Initiating 


Learning 


Acting 
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IEP  Structure 

IEP  Process  Specification 

Candidate 

Common  Team 

Processes 

Team  Member 
Unique  Processes 

Product  Life  Cycle 

Program  Management 

Business  Capture 

Development 

If  applicable 

Production 

If  applicable 

Deployment 

If  applicable 

Operation  &  Sustainment 

If  applicable 

Disposal 

Common  Management 

Planning 

Decision  Analysis 

Configuration  &  Data  Management 

Performance  Assessment  &  Control 

Risk  &  Opportunity  Management 

Indicates  likely  need  for  integrated  team  processes 
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P  View  of  Common  Team 
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■7 


IEP  Structure 

IEP  Process  Specification 

Candidate  Common  Team 
Processes 

Team  Member 
Unique  Processes 

Infrastructure 

Process  Management 

Work  Environment  Management 

Technology  Management 

If  applicable 

Contracts 

Workforce  Management 

Finance 

Supplier  Agreements  &  Procurement 

Security 

Property  Management 

If  applicable  and  co-located 

Enterprise 

Organizational  Management 

Strategic  Planning 

Quality  Management 

Ethics  &  Business  Conduct 

Legal 

Communications 
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Process  Area 
Category 

Process  Area 

Candidate 
Common  Team 

Processes 

Team  Member 
Unique  Processes 

Project  Management 

Project  Planning 

Project  Monitoring  &  Control 

Supplier  Agreement  Management 

Integrated  Project  Management  +  IP  PD 

Risk  Management 

Quantitative  Project  Management 

Engineering 

Requirements  Management 

Requirements  Development 

If  applicable 

Technical  Solution 

If  applicable 

Product  Integration 

Verification 

Validation 

Support 

Configuration  Management 

Process  &  Product  Quality  Assurance 

Measurement  &  Analysis 

Decision  Analysis  &  Resolution 

Causual  Analysis  &  Resolution 

Process  Management 

Organizational  Process  Focus 

Org  =  Program 

Organizational  Process  Definition  +  IPPD 

Org  =  Program 

Organizational  Training 

For  program-specific 

Organizational  Process  Performance 

Organizational  lnno\&tion  &  Deployment 
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•  This  tool  gives  leadership  a  strategic  look 
at  its  value  streams  and  the  ability  to  see 
waste  at  a  macro  level 

•  Leaders  identify  and  prioritize  the 
improvement  events  necessary  to 

■  Eliminate  the  waste 

t  Kaizen  events  (an  activity  where  a  team 
is  chartered  for  a  period  of  3-5  days) 


■  Identify  waste  for  a  given  process  and 

■  Implement  immediate,  sustainable 
solutions  for  waste  elimination/reduction 

t  Other  VSMs,  projects  -  activities  that 
require  extensive  work  and  change  r 

t  Just  Do  Its  -  short  term  projects  u 


The  result:  a  plan  is  in 
place  to  strategically 
identify  and  eliminate  the 
waste  that  most  interferes 
/with  the  ability  to  deliver 
value  to  the  customer. 
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•  Management  Steering  Group 

•  Process  Group 

•  Process  Control  Board 

•  Configuration  Control  Board 

•  Process  Asset  Library 

•  Measurement  Repository 

•  Engineering  Review  Board 


Common  process  infrastructure  at  the  program  level  with 
interfaces  to  team  member-unique  infrastructure 
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Summary 


trade  to 

and  Expanded  Features 


•  Focusing  on  multi-organizational  team  capability  needs  to 
begin  when  teams  are  initially  formed 

•  Deploying  high-capability  processes  early  in  the  life  cycle 
will  enable  multi-organizational  teams  to  reap  benefits  early 

■  Process  discipline  is  key  when  making  programmatic  decisions 
committing  significant  downstream  resources 

•  Ensuring  high-capability  processes  on  multi-organizational 
teams  should  improve  quality  of  products  and  services 

•  Developing  guidance  and  approaches  for  multi- 
organizational  process  team  definition  and  deployment  is 
needed 

•  This  guidance  can  also  be  applied  to  mergers  and 
acquisitions 
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